Implement support for native m8b files #12

tpikonen commented 2025年01月09日 14:07:02 +01:00

The conversion from m8b to msgpack seems to be a source of problems, so add support for the wigle.net m8b files without conversion to msgpack. The file (only the latest one is correctly constructed) works either as-downloaded (with gzip compression) or unpacked. What little documentation there is, is in the example configuration file.

The unpacked file allows much faster queries because they can be memory mapped, but the uncompressed file takes a lot of space. Queries from gzipped files are very slow, but there's a cache which speeds up things considerably after after the first few queries, which likely time out.

I would appreciate some external testing before merging this, @magdesign and @duplexsystem have at least shown interest to this.

The conversion from m8b to msgpack seems to be a source of problems, so add support for the wigle.net m8b files without conversion to msgpack. The file (only the latest one is correctly constructed) works either as-downloaded (with gzip compression) or unpacked. What little documentation there is, is in the example configuration file. The unpacked file allows much faster queries because they can be memory mapped, but the uncompressed file takes a lot of space. Queries from gzipped files are very slow, but there's a cache which speeds up things considerably after after the first few queries, which likely time out. I would appreciate some external testing before merging this, @magdesign and @duplexsystem have at least shown interest to this.

magdesign commented 2025年01月12日 13:51:55 +01:00

its a bit hard to test since the data seems pretty outdated. for the location i am currently in, m8b shows a way further location.... when going online, becondb works because i pushed my location to the database... wondering if taking beacondb offline to the device would help or just going with collecting a local db...

its a bit hard to test since the data seems pretty outdated. for the location i am currently in, m8b shows a way further location.... when going online, becondb works because i pushed my location to the database... wondering if taking beacondb offline to the device would help or just going with collecting a local db...

tpikonen commented 2025年01月13日 17:15:29 +01:00

its a bit hard to test since the data seems pretty outdated.

I've had pretty good results with he latest m8b release here, with only occasionally getting a totally wrong location. The data is old (end of 2022) and somewhat sparse though, so it's probably not as good in most places. Merging this in any case.

> its a bit hard to test since the data seems pretty outdated. I've had pretty good results with he latest m8b release here, with only occasionally getting a totally wrong location. The data is old (end of 2022) and somewhat sparse though, so it's probably not as good in most places. Merging this in any case.