projects:wifi_scanner
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
projects:wifi_scanner [2020/01/06 23:37] neil |
projects:wifi_scanner [2020/01/06 23:53] neil |
||
|---|---|---|---|
| Line 32: | Line 32: | ||
| ===== Importing the data ===== | ===== Importing the data ===== | ||
| - | The raw tcpdump logs are pretty large and full of redundant information - for around a month of wifi scanning it stores around 128 million lines of data (18.6Gb). | + | The raw tcpdump logs are pretty large and full of redundant information - for around a month of wifi scanning it records around 128 million lines of data (18.6Gb). |
| I run the following code to simplify the logs to just pairs of the datetime (in YYYY-MM-DD HH:MM - I strip off the seconds) and the mac address (see below for the php code): | I run the following code to simplify the logs to just pairs of the datetime (in YYYY-MM-DD HH:MM - I strip off the seconds) and the mac address (see below for the php code): | ||
| Line 40: | Line 40: | ||
| </code> | </code> | ||
| - | On my laptop, this processes the log files at around 300k lines/second - so in around 8 minutes. The resulting import file is reduced to approximately x million lines. | + | On my laptop, this processes the log files at around 300k lines/second - so in around 8 minutes. The resulting import file is reduced to approximately 3.8 million lines. |
| I created a simple mysql table to store the timestamp and mac address: | I created a simple mysql table to store the timestamp and mac address: | ||
| Line 50: | Line 50: | ||
| <code sql> | <code sql> | ||
| load data infile 'trimmed_tcpdump.log' into table wifi_data; | load data infile 'trimmed_tcpdump.log' into table wifi_data; | ||
| + | </code> | ||
| + | |||
| + | Once I imported all the data I added an index to the mac address column: | ||
| + | <code sql> | ||
| + | alter table simple_data add index idx_mac(mac); | ||
| </code> | </code> | ||
| Line 55: | Line 60: | ||
| ==== trim.php ==== | ==== trim.php ==== | ||
| - | TBC | + | <code php> |
| + | #!/usr/bin/php | ||
| + | <?php | ||
| + | if(empty($argv[1])) { | ||
| + | exit("Missing filename\n"); | ||
| + | } | ||
| + | $filename = $argv[1]; | ||
| + | $handle = fopen($filename, "r"); | ||
| + | if ($handle) { | ||
| + | while (($line = fgets($handle)) !== false) { | ||
| + | $data = explode(" ", $line); | ||
| + | $datetime = date("Y-m-d H:i", strtotime($data[0]." ".substr($data[1],0, 8))); | ||
| + | $mac_addresses = preg_match_all("/(([a-fA-F0-9]{2}[:|\-]?){6}) /", $line, $matches); | ||
| + | if(is_array($matches[0])) { | ||
| + | $macs = array_unique($matches[0]); | ||
| + | foreach($macs as $mac) { | ||
| + | $mac = trim($mac); | ||
| + | $rawdata[$datetime.$mac]['datetime'] = $datetime; | ||
| + | $rawdata[$datetime.$mac]['mac'] = $mac; | ||
| + | } | ||
| + | } | ||
| + | } | ||
| + | } else { | ||
| + | exit("Error opening file\n"); | ||
| + | } | ||
| + | |||
| + | foreach($rawdata as $datetime=>$val) { | ||
| + | echo $val['datetime']."\t".$val['mac']."\n"; | ||
| + | } | ||
| + | |||
| + | ?> | ||
| + | </code> | ||
| ===== Analysing the data ===== | ===== Analysing the data ===== | ||
projects/wifi_scanner.txt · Last modified: 2020/08/03 16:11 by admin
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Share Alike 4.0 International
