Differences

This shows you the differences between two versions of the page.

--- projects:wifi_scanner [2019/10/02 14:50]
neil [Analysing the data]
+++ projects:wifi_scanner [2020/01/06 23:31]
neil
@@ Line 14: / Line 14: @@
 ==== channel_changer.php ====
 <code php>
-?php
+#!/usr/bin/php
+<?php
 $channels = array(
 ,2,3,4,5,6,7,8,9,10,11, 12,13,36,40,44,48,52,56,60,64,100,104,
@@ Line 29: / Line 30: @@
 ?>
 </code>
+===== Importing the data =====
+I import the raw tcpdump logs (just a timestamp and mac address) into a simple mysql table:
+<code sql>
+create table wifi_data (seen_time datetime, mac varchar(17), unique (seen_time,mac));
+</code>
+These files are pretty large - for around a month of wifi scanning data is around 128.4 million lines of data (18.6Gb).  I run the following code to simplify the logs to just pairs of the datetime (in YYYY-MM-DD HH:MM) and the mac address:
+<code bash>
+php trim.php tcpdump.log > trimmed_tcpdump.log
+</code>
+This takes around 12 minutes which reduces the number of lines of data to around 20 million.  Then I import this directly to the mysql database using the mysql client:
+<code sql>
+load data infile 'trimmed_tcpdump.log' into table wifi_data;
+</code>
+If you have any trouble with this command, you might want to split the file into more managble parts using ''split -l 1000000 trimmed_tcpdump.log''
+==== trim.php ====
+TBC
 ===== Analysing the data =====
 I've made some graphs:
-  * [[https://starflyer.armchairscientist.co.uk/tmp/wifi3.php|General scan #2 - Sat July 6th 2019 8am-11:30am]] - All data grouped in unique MACs per 5 minute period
+  * [[https://starflyer.armchairscientist.co.uk/tmp/wifi3.php|General scan #2 - Sat July 6th 2019 8am-11:30am]] - All data grouped in unique MACs per 1 minute period
-  * [[https://starflyer.armchairscientist.co.uk/tmp/wifi3.php|General scan #3 - Sat July 6th 2019 8am-11:30am]] - As above, known devices/equipment filtered - an example of identifying a group of passers (Orange walk outside my window at 10:40am)
+  * [[https://starflyer.armchairscientist.co.uk/tmp/wifi3.php|General scan #3 - Sat July 6th 2019 8am-11:30am]] - As above, known (previously seen the hours/days before) devices/equipment filtered - an example of identifying a group of passerbys (Orange walk outside my window at 10:40am)
   * [[https://starflyer.armchairscientist.co.uk/tmp/wifi4.php|General scan #4]] - Multiple days showing the weekend and spikes, at 8am and 5pm, of people passing by to and from work.

neil.mckillop.org

User Tools

Site Tools

Differences

Page Tools