Traffic: crashes while routing #2937

mvglasow commented 2025年12月20日 18:51:56 +01:00

I am getting sporadic crashes while routing, seemingly out of the blue.

No logcat yet, as these crashes always happened when I was busy driving.

I have not yet managed to reproduce them on an official build to rule out traffic features as the cause.

Tested on Android (LineageOS 20) with 426a9661 (traffic with d473361e5 merged).

Device is a Fairphone 4, which has a known bug where phantom touch events occur. Asides from Traffic, this is another possible cause of the bug (the thunderstorm of touch events which occasionally rains down on the screen and triggers random functions might upset the graphics engine or other components).

I am getting sporadic crashes while routing, seemingly out of the blue. No logcat yet, as these crashes always happened when I was busy driving. I have not yet managed to reproduce them on an official build to rule out traffic features as the cause. Tested on Android (LineageOS 20) with 426a9661 (traffic with d473361e5 merged). Device is a Fairphone 4, which has a known bug where phantom touch events occur. Asides from Traffic, this is another possible cause of the bug (the thunderstorm of touch events which occasionally rains down on the screen and triggers random functions might upset the graphics engine or other components).

zyphlar commented 2025年12月20日 19:22:18 +01:00

IIRC you can just leave logging on indefinitely. I'm not sure if it prunes old logs or what but I'll tend to leave it on and it seems fine

IIRC you can just leave logging on indefinitely. I'm not sure if it prunes old logs or what but I'll tend to leave it on and it seems fine

MXC480 commented 2025年12月26日 15:19:20 +01:00

Where do you download this traffic version?? I want to test here :)

Where do you download this traffic version?? I want to test here :)

jeanbaptisteC commented 2025年12月26日 15:27:03 +01:00

Feature is not yet available, be patient 😉

Feature is not yet available, be patient 😉

mvglasow commented 2026年01月11日 14:07:07 +01:00

Yep, the log gets written to Android/data/app.comaps.*/files/logs/app.log. However, it gets cleared on app restart, which is a bummer if either the app restarts by itself, or if you restart it manually before copying away the log. If the app crashes while on the road, your options are to either go without navigation until you can pull over, make a backup of the log and only then restart the app, or restart the app instantly and lose all log information.

Because of that, I no longer have a log of the crash, I need to wait for another crash.

Also, the interesting question is how much gets logged in case of a crash – if the app gets killed before it manages to write its last log entries, that would be a bummer.

I noticed two things: no crashes in the official build, only the traffic branch is affected. Also, crashes seem to occur more frequently when using Qz with a TMC receiver – I suspect that is because messages arrive continuously, and information is more dynamic: locations change as queues grow or shrink, or the vehicle moves to a different reception area where it receives entirely new messages. The traffic server is somewhat more static in that respect. All of this points to the crash being somehow related to traffic, specifically message decoding or route recalculation.

Yep, the log gets written to `Android/data/app.comaps.*/files/logs/app.log`. However, it gets cleared on app restart, which is a bummer if either the app restarts by itself, or if you restart it manually before copying away the log. If the app crashes while on the road, your options are to either go without navigation until you can pull over, make a backup of the log and only then restart the app, or restart the app instantly and lose all log information. Because of that, I no longer have a log of the crash, I need to wait for another crash. Also, the interesting question is how much gets logged in case of a crash – if the app gets killed before it manages to write its last log entries, that would be a bummer. I noticed two things: no crashes in the official build, only the traffic branch is affected. Also, crashes seem to occur more frequently when using Qz with a TMC receiver – I suspect that is because messages arrive continuously, and information is more dynamic: locations change as queues grow or shrink, or the vehicle moves to a different reception area where it receives entirely new messages. The traffic server is somewhat more static in that respect. All of this points to the crash being somehow related to traffic, specifically message decoding or route recalculation.

mvglasow commented 2026年01月11日 15:08:34 +01:00

Just had a crash in the desktop version which might be related, though I’m not 100% sure. I had calculated a long route, and traffic was still decoding. Log output here:

I(2) 1888.35669 map/traffic_manager.cpp:723 DecodeFirstMessage(): de.viznrw.traffic:r:LMS-BY/r_LMS-BY/1264375_D BY LMS-BY.1 : TraffMessage { id: de.viznrw.traffic:r:LMS-BY/r_LMS-BY/1264375_D BY LMS-BY.1, replaces: [ ], receiveTime: 2026年01月06日 05:03:44 UTC, updateTime: 2026年01月11日 06:13:40 UTC, expirationTime: 2026年01月11日 06:15:38 UTC, startTime: nullopt, endTime: nullopt, cancellation: 1, forecast: 0, location: nullopt, events: [ ] }
I(2) 1888.35755 map/traffic_manager.cpp:830 WaitForRequest(): feed queue not empty, returning immediately
I(2) 1888.35760 map/traffic_manager.cpp:781 ThreadRoutine(): active MWMs changed: 0 , poll needed: 0
I(2) 1888.35763 map/traffic_manager.cpp:797 ThreadRoutine(): 1 feed(s) in queue
I(2) 1888.39312 map/traffic_manager.cpp:718 DecodeFirstMessage(): message cz.rsd.commonti:s:294cfc48-c250-4451-b1e5-1190da242856 is already in cache, skipping
I(2) 1888.39318 map/traffic_manager.cpp:830 WaitForRequest(): feed queue not empty, returning immediately
I(2) 1888.39322 map/traffic_manager.cpp:781 ThreadRoutine(): active MWMs changed: 0 , poll needed: 0
I(2) 1888.39324 map/traffic_manager.cpp:797 ThreadRoutine(): 1 feed(s) in queue
I(1) 1888.42860 map/traffic_manager.cpp:213 OnChangeRoutingSessionState(): Routing session state changed from RouteRebuilding to RouteNotStarted
I(1) 1888.42865 map/traffic_manager.cpp:214 OnChangeRoutingSessionState(): Running on thread 136539091856576
I(1) 1888.42874 map/traffic_manager.cpp:248 OnChangeRoutingSessionState(): Router MWMs: [17: Austria_Salzburg Austria_Tyrol Austria_Upper Austria_Linz Austria_Upper Austria_Wels Czech_Jihovychod_Kraj Vysocina Czech_Jihozapad_Jihocesky kraj Czech_Severovychod_Kralovehradecky kraj Czech_Severovychod_Pardubicky kraj Germany_Free State of Bavaria_Upper Bavaria_East Lithuania_East Lithuania_West Poland_Greater Poland Voivodeship Poland_Lodz Voivodeship Poland_Lower Silesian Voivodeship Poland_Masovian Voivodeship Poland_Podlaskie Voivodeship Poland_Warmian-Masurian Voivodeship ]
Segmentation fault (core dumped)

Just had a crash in the desktop version which might be related, though I’m not 100% sure. I had calculated a long route, and traffic was still decoding. Log output here: ``` I(2) 1888.35669 map/traffic_manager.cpp:723 DecodeFirstMessage(): de.viznrw.traffic:r:LMS-BY/r_LMS-BY/1264375_D BY LMS-BY.1 : TraffMessage { id: de.viznrw.traffic:r:LMS-BY/r_LMS-BY/1264375_D BY LMS-BY.1, replaces: [ ], receiveTime: 2026年01月06日 05:03:44 UTC, updateTime: 2026年01月11日 06:13:40 UTC, expirationTime: 2026年01月11日 06:15:38 UTC, startTime: nullopt, endTime: nullopt, cancellation: 1, forecast: 0, location: nullopt, events: [ ] } I(2) 1888.35755 map/traffic_manager.cpp:830 WaitForRequest(): feed queue not empty, returning immediately I(2) 1888.35760 map/traffic_manager.cpp:781 ThreadRoutine(): active MWMs changed: 0 , poll needed: 0 I(2) 1888.35763 map/traffic_manager.cpp:797 ThreadRoutine(): 1 feed(s) in queue I(2) 1888.39312 map/traffic_manager.cpp:718 DecodeFirstMessage(): message cz.rsd.commonti:s:294cfc48-c250-4451-b1e5-1190da242856 is already in cache, skipping I(2) 1888.39318 map/traffic_manager.cpp:830 WaitForRequest(): feed queue not empty, returning immediately I(2) 1888.39322 map/traffic_manager.cpp:781 ThreadRoutine(): active MWMs changed: 0 , poll needed: 0 I(2) 1888.39324 map/traffic_manager.cpp:797 ThreadRoutine(): 1 feed(s) in queue I(1) 1888.42860 map/traffic_manager.cpp:213 OnChangeRoutingSessionState(): Routing session state changed from RouteRebuilding to RouteNotStarted I(1) 1888.42865 map/traffic_manager.cpp:214 OnChangeRoutingSessionState(): Running on thread 136539091856576 I(1) 1888.42874 map/traffic_manager.cpp:248 OnChangeRoutingSessionState(): Router MWMs: [17: Austria_Salzburg Austria_Tyrol Austria_Upper Austria_Linz Austria_Upper Austria_Wels Czech_Jihovychod_Kraj Vysocina Czech_Jihozapad_Jihocesky kraj Czech_Severovychod_Kralovehradecky kraj Czech_Severovychod_Pardubicky kraj Germany_Free State of Bavaria_Upper Bavaria_East Lithuania_East Lithuania_West Poland_Greater Poland Voivodeship Poland_Lodz Voivodeship Poland_Lower Silesian Voivodeship Poland_Masovian Voivodeship Poland_Podlaskie Voivodeship Poland_Warmian-Masurian Voivodeship ] Segmentation fault (core dumped) ```

mvglasow commented 2026年01月11日 16:39:33 +01:00

And another one:

I(2) 5278.12447 map/traffic_manager.cpp:723 DecodeFirstMessage(): lt.eismoinfo.trafficintensity:1885,p : TraffMessage { id: lt.eismoinfo.trafficintensity:1885,p, replaces: [ ], receiveTime: 2026年01月02日 00:00:00 UTC, updateTime: 2026年01月11日 15:00:00 UTC, expirationTime: 2026年01月11日 15:32:00 UTC, startTime: nullopt, endTime: nullopt, cancellation: 0, forecast: 0, location: TraffLocation { from: Point { coordinates: ms::LatLon(55.223522, 24.863277), distance: nullopt, junctionName: nullopt, junctionRef: nullopt }, at: nullopt, via: nullopt, to: Point { coordinates: ms::LatLon(55.224251, 24.847614), distance: 6.760000, junctionName: Paprėniškiai, junctionRef: nullopt }, notVia: nullopt, fuzziness: nullopt, country: LT, territory: nullopt, town: nullopt, roadClass: Primary, roadRef: 115, roadName: nullopt, origin: Molėtai, destination: Ukmergė, direction: nullopt, directionality: OneDirection, ramps: None }, events: [ TraffEvent { class: Congestion, type: CongestionNormalTraffic, length: nullopt, probability: nullopt, q_duration: nullopt, speed: 87 } ] }
I(2) 5278.12915 traffxml/traff_decoder.cpp:257 DecodeMessage(): Impact: optional(TrafficImpact { speedGroup: G5, maxspeed: 87, delayMins: 0 })
I(2) 5278.12939 traffxml/traff_decoder.cpp:1107 DecodeLocationDirection(): Calculating the route of direct length 997.782 m. checkpoints: Checkpoints(ms::LatLon(55.223522, 24.863277); ms::LatLon(55.224251, 24.847614); passed: 0) startDirection: m2::Point<d>(0, 0) router name: astar-bidirectional-Car
I(2) 5278.13102 routing/index_router.cpp:1106 MakeWorldGraph(): Avoid next roads: RoutingOptions: {}
I(8) 5279.07172 routing/async_router.cpp:215 LogCode(): Route found, elapsed seconds: 134.564
I(8) 5279.07180 routing/async_router.cpp:323 CalculateRoute(): ETA: 60613.8 sec.
I(1) 5279.09004 map/traffic_manager.cpp:213 OnChangeRoutingSessionState(): Routing session state changed from RouteRebuilding to RouteNotStarted
I(1) 5279.09012 map/traffic_manager.cpp:214 OnChangeRoutingSessionState(): Running on thread 129651287710912
I(1) 5279.09019 map/traffic_manager.cpp:248 OnChangeRoutingSessionState(): Router MWMs: [17: Austria_Salzburg Austria_Tyrol Austria_Upper Austria_Linz Austria_Upper Austria_Wels Czech_Jihovychod_Kraj Vysocina Czech_Jihozapad_Jihocesky kraj Czech_Severovychod_Kralovehradecky kraj Czech_Severovychod_Pardubicky kraj Germany_Free State of Bavaria_Upper Bavaria_East Lithuania_East Lithuania_West Poland_Greater Poland Voivodeship Poland_Lodz Voivodeship Poland_Lower Silesian Voivodeship Poland_Masovian Voivodeship Poland_Podlaskie Voivodeship Poland_Warmian-Masurian Voivodeship ]
I(1) 5279.09028 map/traffic_manager.cpp:258 OnChangeRoutingSessionState(): MWM set: [17: MwmId [Austria_Upper Austria_Linz, 250804] MwmId [Czech_Jihovychod_Kraj Vysocina, 250804] MwmId [Austria_Salzburg, 250804] MwmId [Poland_Podlaskie Voivodeship, 250804] MwmId [Poland_Masovian Voivodeship, 250804] MwmId [Czech_Severovychod_Pardubicky kraj, 250804] MwmId [Austria_Tyrol, 250804] MwmId [Poland_Lodz Voivodeship, 250804] MwmId [Poland_Lower Silesian Voivodeship, 250804] MwmId [Lithuania_West, 250804] MwmId [Poland_Greater Poland Voivodeship, 250804] MwmId [Germany_Free State of Bavaria_Upper Bavaria_East, 250804] MwmId [Czech_Severovychod_Kralovehradecky kraj, 250804] MwmId [Lithuania_East, 250804] MwmId [Poland_Warmian-Masurian Voivodeship, 250804] MwmId [Czech_Jihozapad_Jihocesky kraj, 250804] MwmId [Austria_Upper Austria_Wels, 250804] ]
Segmentation fault (core dumped)

So the crash likely happens during OnChangeRoutingSessionState(). In the first case, it got to at least to line 248; in the second case it ran beyond line 248 up to at least line 258. We might be looking at a race condition involving unsynchronized access to the same data here.

And another one: ``` I(2) 5278.12447 map/traffic_manager.cpp:723 DecodeFirstMessage(): lt.eismoinfo.trafficintensity:1885,p : TraffMessage { id: lt.eismoinfo.trafficintensity:1885,p, replaces: [ ], receiveTime: 2026年01月02日 00:00:00 UTC, updateTime: 2026年01月11日 15:00:00 UTC, expirationTime: 2026年01月11日 15:32:00 UTC, startTime: nullopt, endTime: nullopt, cancellation: 0, forecast: 0, location: TraffLocation { from: Point { coordinates: ms::LatLon(55.223522, 24.863277), distance: nullopt, junctionName: nullopt, junctionRef: nullopt }, at: nullopt, via: nullopt, to: Point { coordinates: ms::LatLon(55.224251, 24.847614), distance: 6.760000, junctionName: Paprėniškiai, junctionRef: nullopt }, notVia: nullopt, fuzziness: nullopt, country: LT, territory: nullopt, town: nullopt, roadClass: Primary, roadRef: 115, roadName: nullopt, origin: Molėtai, destination: Ukmergė, direction: nullopt, directionality: OneDirection, ramps: None }, events: [ TraffEvent { class: Congestion, type: CongestionNormalTraffic, length: nullopt, probability: nullopt, q_duration: nullopt, speed: 87 } ] } I(2) 5278.12915 traffxml/traff_decoder.cpp:257 DecodeMessage(): Impact: optional(TrafficImpact { speedGroup: G5, maxspeed: 87, delayMins: 0 }) I(2) 5278.12939 traffxml/traff_decoder.cpp:1107 DecodeLocationDirection(): Calculating the route of direct length 997.782 m. checkpoints: Checkpoints(ms::LatLon(55.223522, 24.863277); ms::LatLon(55.224251, 24.847614); passed: 0) startDirection: m2::Point<d>(0, 0) router name: astar-bidirectional-Car I(2) 5278.13102 routing/index_router.cpp:1106 MakeWorldGraph(): Avoid next roads: RoutingOptions: {} I(8) 5279.07172 routing/async_router.cpp:215 LogCode(): Route found, elapsed seconds: 134.564 I(8) 5279.07180 routing/async_router.cpp:323 CalculateRoute(): ETA: 60613.8 sec. I(1) 5279.09004 map/traffic_manager.cpp:213 OnChangeRoutingSessionState(): Routing session state changed from RouteRebuilding to RouteNotStarted I(1) 5279.09012 map/traffic_manager.cpp:214 OnChangeRoutingSessionState(): Running on thread 129651287710912 I(1) 5279.09019 map/traffic_manager.cpp:248 OnChangeRoutingSessionState(): Router MWMs: [17: Austria_Salzburg Austria_Tyrol Austria_Upper Austria_Linz Austria_Upper Austria_Wels Czech_Jihovychod_Kraj Vysocina Czech_Jihozapad_Jihocesky kraj Czech_Severovychod_Kralovehradecky kraj Czech_Severovychod_Pardubicky kraj Germany_Free State of Bavaria_Upper Bavaria_East Lithuania_East Lithuania_West Poland_Greater Poland Voivodeship Poland_Lodz Voivodeship Poland_Lower Silesian Voivodeship Poland_Masovian Voivodeship Poland_Podlaskie Voivodeship Poland_Warmian-Masurian Voivodeship ] I(1) 5279.09028 map/traffic_manager.cpp:258 OnChangeRoutingSessionState(): MWM set: [17: MwmId [Austria_Upper Austria_Linz, 250804] MwmId [Czech_Jihovychod_Kraj Vysocina, 250804] MwmId [Austria_Salzburg, 250804] MwmId [Poland_Podlaskie Voivodeship, 250804] MwmId [Poland_Masovian Voivodeship, 250804] MwmId [Czech_Severovychod_Pardubicky kraj, 250804] MwmId [Austria_Tyrol, 250804] MwmId [Poland_Lodz Voivodeship, 250804] MwmId [Poland_Lower Silesian Voivodeship, 250804] MwmId [Lithuania_West, 250804] MwmId [Poland_Greater Poland Voivodeship, 250804] MwmId [Germany_Free State of Bavaria_Upper Bavaria_East, 250804] MwmId [Czech_Severovychod_Kralovehradecky kraj, 250804] MwmId [Lithuania_East, 250804] MwmId [Poland_Warmian-Masurian Voivodeship, 250804] MwmId [Czech_Jihozapad_Jihocesky kraj, 250804] MwmId [Austria_Upper Austria_Wels, 250804] ] Segmentation fault (core dumped) ``` So the crash likely happens during `OnChangeRoutingSessionState()`. In the first case, it got to at least to line 248; in the second case it ran beyond line 248 up to at least line 258. We might be looking at a race condition involving unsynchronized access to the same data here.

mvglasow commented 2026年02月21日 21:10:42 +01:00

Help wanted: Symptoms indicate unsynchronized access, but I have no idea which threads and data structures are involved. We are talking about a very sporadic error which may take some time to reproduce. Helgrind slows down the code massively (rendering it near-unusable). Even in theory, without proper IDE integration I have no idea how to translate addresses from Helgrind into data structures in the code. Can anyone help with these things?

Help wanted: Symptoms indicate unsynchronized access, but I have no idea which threads and data structures are involved. We are talking about a very sporadic error which may take some time to reproduce. Helgrind slows down the code massively (rendering it near-unusable). Even in theory, without proper IDE integration I have no idea how to translate addresses from Helgrind into data structures in the code. Can anyone help with these things?

mvglasow commented 2026年02月22日 09:39:55 +01:00

For the crash between lines 248–258, this is the affected code:

LOG(LINFO, ("Router MWMs:", mwmNames));
std::set<MwmSet::MwmId> mwms;
for (auto const & mwmName : mwmNames)
{
 MwmSet::MwmId mwmId = m_dataSource.GetMwmIdByCountryFile(platform::CountryFile(mwmName));
 if (mwmId.IsAlive())
 mwms.insert(mwmId);
}
LOG(LINFO, ("MWM set:", mwms));

The only operation on non-local variables is the call to m_dataSource.GetMwmIdByCountryFile(). Though we should only be performing read operations here, and spontaneously I can’t think of any reason any other thread would be attempting a write operation on related structures at the same time. (Write operations would be most likely to occur when loading, unloading, adding or updating MWMs, neither of which should have happened at that time (CoMaps had fully started up and no map downloads were active).

For the crash between lines 248–258, this is the affected code: LOG(LINFO, ("Router MWMs:", mwmNames)); std::set<MwmSet::MwmId> mwms; for (auto const & mwmName : mwmNames) { MwmSet::MwmId mwmId = m_dataSource.GetMwmIdByCountryFile(platform::CountryFile(mwmName)); if (mwmId.IsAlive()) mwms.insert(mwmId); } LOG(LINFO, ("MWM set:", mwms)); The only operation on non-local variables is the call to `m_dataSource.GetMwmIdByCountryFile()`. Though we should only be performing read operations here, and spontaneously I can’t think of any reason any other thread would be attempting a write operation on related structures at the same time. (Write operations would be most likely to occur when loading, unloading, adding or updating MWMs, neither of which should have happened at that time (CoMaps had fully started up and no map downloads were active).

mvglasow commented 2026年02月22日 10:39:07 +01:00

Plus, GetMwmIdByCountryFile() calls locks a mutex before calling its implementation, which is an argument against it being the culprit.

Plus, `GetMwmIdByCountryFile()` calls locks a mutex before calling its implementation, which is an argument against it being the culprit.

mvglasow commented 2026年02月22日 11:42:25 +01:00

Output from another crash, this time past line 258, captured with gdb -q -batch -ex "run" -ex "bt" --args ~/src/omim-build-release/CoMaps:

I(1) 204.27330 map/traffic_manager.cpp:213 OnChangeRoutingSessionState(): Routing session state changed from RouteRebuilding to RouteNotStarted
I(1) 204.27337 map/traffic_manager.cpp:214 OnChangeRoutingSessionState(): Running on thread 140737253072064
I(1) 204.27613 map/traffic_manager.cpp:248 OnChangeRoutingSessionState(): Router MWMs: [16: Czech_Jihozapad_Jihocesky kraj Czech_Severovychod_Liberecky kraj Czech_Stredni Cechy_East Czech_Stredni Cechy_West Germany_Free State of Bavaria_Lower Bavaria Germany_Free State of Bavaria_Upper Bavaria_East Germany_Free State of Bavaria_Upper Bavaria_Munchen Lithuania_East Lithuania_West Poland_Greater Poland Voivodeship Poland_Kuyavian-Pomeranian Voivodeship Poland_Lower Silesian Voivodeship Poland_Lubusz Voivodeship Poland_Warmian-Masurian Voivodeship Russia_Kaliningrad Oblast ���[��WW��3W�oland Voivodeship ]
I(1) 204.27621 map/traffic_manager.cpp:258 OnChangeRoutingSessionState(): MWM set: [14: MwmId [Poland_Greater Poland Voivodeship, 251217] MwmId [Poland_Kuyavian-Pomeranian Voivodeship, 251217] MwmId [Czech_Jihozapad_Jihocesky kraj, 251217] MwmId [Poland_Warmian-Masurian Voivodeship, 251217] MwmId [Poland_Lower Silesian Voivodeship, 251217] MwmId [Lithuania_East, 251217] MwmId [Lithuania_West, 251217] MwmId [Germany_Free State of Bavaria_Lower Bavaria, 251217] MwmId [Poland_Lubusz Voivodeship, 251217] MwmId [Czech_Stredni Cechy_West, 251217] MwmId [Czech_Severovychod_Liberecky kraj, 251217] MwmId [Germany_Free State of Bavaria_Upper Bavaria_Munchen, 251217] MwmId [Germany_Free State of Bavaria_Upper Bavaria_East, 251217] MwmId [Czech_Stredni Cechy_East, 251217] ]
I(2) 204.27626 map/traffic_manager.cpp:869 WaitForRequest(): timeout: 0 active MWMs changed: 1 test mode: 0
I(2) 204.27628 map/traffic_manager.cpp:781 ThreadRoutine(): active MWMs changed: 1 , poll needed: 0
Thread 13 "CoMaps" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffca7fc6c0 (LWP 107544)]
0x000055555590d864 in routing::AbsentRegionsFinder::GetAllRegions(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) ()
#0 0x000055555590d864 in routing::AbsentRegionsFinder::GetAllRegions(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) ()
#1 0x000055555590f40e in routing::AsyncRouter::CalculateRoute() ()
#2 0x000055555590fd2e in routing::AsyncRouter::ThreadFunc() ()
#3 0x0000555555c791d2 in threads::SimpleThread::ThreadFunc(std::function<void ()>&&) ()
#4 0x0000555555921d42 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::function<void ()>&&), std::_Bind<void (routing::AsyncRouter::*(routing::AsyncRouter*))()> > > >::_M_run() ()
#5 0x00007ffff5aecdb4 in ??? () at /lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff569caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#7 0x00007ffff5729c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

The thread that triggered the segmentation fault is not the thread in which OnChangeRoutingSessionState() runs (see second line of log output and , which explains why log output differs each time. (This time we even have log output beyond line 258.)

Looking at the stack trace, I would suspect the thread that crashes to be some router thread (either the router thread, or a router thread spawned by the decoder – need to analyze the inner workings of the decoder to tell for sure).

The function in which the segfault occurred is routing::AbsentRegionsFinder::GetAllRegions().

I am also seeing some corruption on the other thread in the log output (at 204.27613), not sure if it is relevant here.

Output from another crash, this time past line 258, captured with `gdb -q -batch -ex "run" -ex "bt" --args ~/src/omim-build-release/CoMaps`: ``` I(1) 204.27330 map/traffic_manager.cpp:213 OnChangeRoutingSessionState(): Routing session state changed from RouteRebuilding to RouteNotStarted I(1) 204.27337 map/traffic_manager.cpp:214 OnChangeRoutingSessionState(): Running on thread 140737253072064 I(1) 204.27613 map/traffic_manager.cpp:248 OnChangeRoutingSessionState(): Router MWMs: [16: Czech_Jihozapad_Jihocesky kraj Czech_Severovychod_Liberecky kraj Czech_Stredni Cechy_East Czech_Stredni Cechy_West Germany_Free State of Bavaria_Lower Bavaria Germany_Free State of Bavaria_Upper Bavaria_East Germany_Free State of Bavaria_Upper Bavaria_Munchen Lithuania_East Lithuania_West Poland_Greater Poland Voivodeship Poland_Kuyavian-Pomeranian Voivodeship Poland_Lower Silesian Voivodeship Poland_Lubusz Voivodeship Poland_Warmian-Masurian Voivodeship Russia_Kaliningrad Oblast ���[��WW��3W�oland Voivodeship ] I(1) 204.27621 map/traffic_manager.cpp:258 OnChangeRoutingSessionState(): MWM set: [14: MwmId [Poland_Greater Poland Voivodeship, 251217] MwmId [Poland_Kuyavian-Pomeranian Voivodeship, 251217] MwmId [Czech_Jihozapad_Jihocesky kraj, 251217] MwmId [Poland_Warmian-Masurian Voivodeship, 251217] MwmId [Poland_Lower Silesian Voivodeship, 251217] MwmId [Lithuania_East, 251217] MwmId [Lithuania_West, 251217] MwmId [Germany_Free State of Bavaria_Lower Bavaria, 251217] MwmId [Poland_Lubusz Voivodeship, 251217] MwmId [Czech_Stredni Cechy_West, 251217] MwmId [Czech_Severovychod_Liberecky kraj, 251217] MwmId [Germany_Free State of Bavaria_Upper Bavaria_Munchen, 251217] MwmId [Germany_Free State of Bavaria_Upper Bavaria_East, 251217] MwmId [Czech_Stredni Cechy_East, 251217] ] I(2) 204.27626 map/traffic_manager.cpp:869 WaitForRequest(): timeout: 0 active MWMs changed: 1 test mode: 0 I(2) 204.27628 map/traffic_manager.cpp:781 ThreadRoutine(): active MWMs changed: 1 , poll needed: 0 Thread 13 "CoMaps" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffca7fc6c0 (LWP 107544)] 0x000055555590d864 in routing::AbsentRegionsFinder::GetAllRegions(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () #0 0x000055555590d864 in routing::AbsentRegionsFinder::GetAllRegions(std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) () #1 0x000055555590f40e in routing::AsyncRouter::CalculateRoute() () #2 0x000055555590fd2e in routing::AsyncRouter::ThreadFunc() () #3 0x0000555555c791d2 in threads::SimpleThread::ThreadFunc(std::function<void ()>&&) () #4 0x0000555555921d42 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::function<void ()>&&), std::_Bind<void (routing::AsyncRouter::*(routing::AsyncRouter*))()> > > >::_M_run() () #5 0x00007ffff5aecdb4 in ??? () at /lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x00007ffff569caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #7 0x00007ffff5729c6c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 ``` The thread that triggered the segmentation fault is not the thread in which `OnChangeRoutingSessionState()` runs (see second line of log output and , which explains why log output differs each time. (This time we even have log output beyond line 258.) Looking at the stack trace, I would suspect the thread that crashes to be some router thread (either **the** router thread, or a router thread spawned by the decoder – need to analyze the inner workings of the decoder to tell for sure). The function in which the segfault occurred is `routing::AbsentRegionsFinder::GetAllRegions()`. I am also seeing some corruption on the other thread in the log output (at 204.27613), not sure if it is relevant here.

mvglasow commented 2026年02月22日 13:03:53 +01:00

AsyncRouter::ThreadFunc() is the thread routine for AsyncRouter::m_thread, which is initialized in the AsyncRouter constructor.

AsyncRouter is a wrapper class to run routing routines in a different thread. It encapsulates an IRouter (or subclass) instance and runs it in a separate worker thread to calculate the route.

RoutingTraffDecoder encapsulates an IRouter directly, running the router on the traffic worker thread.

All of this indicates the segfault occurs on the router thread.

Since I remember there being some issues with crashes while routing, I did some research and the only one I found was #2131. However, it does not look related.

I do notice that routing::AbsentRegionsFinder::GetAllRegions() is indirectly called from two places (not counting tests):

routing::AbsentRegionsFinder::GetAllRegions(...)
 routing::AbsentRegionsFinder::GetAbsentRegions(...)
 routing::AsyncRouter::CalculateRoute() // without arguments
 routing::AsyncRouter::GetAllRegions(...)
 routing::RoutingSession::GetAllRegions(...)
 TrafficManager::OnChangeRoutingSessionState(...)

So the routing thread is inside routing::AsyncRouter::GetAllRegions(), and the UI thread is inside a function which may also call routing::AsyncRouter::GetAllRegions().

That call happens in traffic_manager.cpp:241. The two functions in between in the call stack do nothing apart from verifying that the class instance is not null before calling its method.

Th next place where something happens is routing::AsyncRouter::GetAllRegions(). It checks if it has a router thread, which should only be the case when called from the router. If so, it retrieves the regions from the thread and caches them. If not, it returns the cached regions.

The corruption in the set of MWMs I am seeing above indicates unsynchronized access to the list of regions. Indeed, the structure is not protected in any way. So I guess that’s the error.

`AsyncRouter::ThreadFunc()` is the thread routine for `AsyncRouter::m_thread`, which is initialized in the `AsyncRouter` constructor. `AsyncRouter` is a wrapper class to run routing routines in a different thread. It encapsulates an `IRouter` (or subclass) instance and runs it in a separate worker thread to calculate the route. `RoutingTraffDecoder` encapsulates an `IRouter` directly, running the router on the traffic worker thread. All of this indicates the segfault occurs on the router thread. Since I remember there being some issues with crashes while routing, I did some research and the only one I found was #2131. However, it does not look related. I do notice that `routing::AbsentRegionsFinder::GetAllRegions()` is indirectly called from two places (not counting tests): ``` routing::AbsentRegionsFinder::GetAllRegions(...) routing::AbsentRegionsFinder::GetAbsentRegions(...) routing::AsyncRouter::CalculateRoute() // without arguments routing::AsyncRouter::GetAllRegions(...) routing::RoutingSession::GetAllRegions(...) TrafficManager::OnChangeRoutingSessionState(...) ``` So the routing thread is inside `routing::AsyncRouter::GetAllRegions()`, and the UI thread is inside a function which may also call `routing::AsyncRouter::GetAllRegions()`. That call happens in `traffic_manager.cpp:241`. The two functions in between in the call stack do nothing apart from verifying that the class instance is not null before calling its method. Th next place where something happens is `routing::AsyncRouter::GetAllRegions()`. It checks if it has a router thread, which should only be the case when called from the router. If so, it retrieves the regions from the thread and caches them. If not, it returns the cached regions. The corruption in the set of MWMs I am seeing above indicates unsynchronized access to the list of regions. Indeed, the structure is not protected in any way. So I guess that’s the error.

mvglasow commented 2026年02月22日 13:36:15 +01:00

Fixed synchronization in 10ed4682c (turns out the mutex was already there, comments and everything, I just forgot to actually lock it before accessing the member it was meant to protect). Currently running some lab tests. Closing this for now, with the option to reopen, should it turn out that issues still persist (due to the sporadic nature of the bug, it is hard to prove with certainty that it is resolved).

Fixed synchronization in 10ed4682c (turns out the mutex was already there, comments and everything, I just forgot to actually lock it before accessing the member it was meant to protect). Currently running some lab tests. Closing this for now, with the option to reopen, should it turn out that issues still persist (due to the sporadic nature of the bug, it is hard to prove with certainty that it is resolved).

mvglasow commented 2026年02月22日 14:00:43 +01:00

Still getting segfaults in routing::AbsentRegionsFinder::GetAllRegions(), need to investigate further.

Still getting segfaults in `routing::AbsentRegionsFinder::GetAllRegions()`, need to investigate further.

mvglasow commented 2026年02月22日 14:31:36 +01:00

Not 100% sure if that is an issue, but access to AbsentRegionsFinder::m_routerThread is also unsynchronized.

Just to be sure, I am protecting the entire function bodies of GenerateAbsentRegions() and GetAllRegions() with the mutex. That would protect access to any and all non-const members of AbsentRegionsFinder – at the cost of potentially locking up the UI thread, should it call GetAllRegions() while the router is also calling it (and waiting for the router thread to finish).

The only way for the UI thread to call GetAllRegions() is from TrafficManager::OnChangeRoutingSessionState(), which only happens for certain state transitions – namely those which indicate the router (and thus the router thread) has finished. So I’m not expecting that to cause any issues in practice.

Not 100% sure if that is an issue, but access to `AbsentRegionsFinder::m_routerThread` is also unsynchronized. Just to be sure, I am protecting the entire function bodies of `GenerateAbsentRegions()` and `GetAllRegions()` with the mutex. That would protect access to any and all non-`const` members of `AbsentRegionsFinder` – at the cost of potentially locking up the UI thread, should it call `GetAllRegions()` while the router is also calling it (and waiting for the router thread to finish). The only way for the UI thread to call `GetAllRegions()` is from `TrafficManager::OnChangeRoutingSessionState()`, which only happens for certain state transitions – namely those which indicate the router (and thus the router thread) has finished. So I’m not expecting that to cause any issues in practice.

mvglasow commented 2026年02月22日 15:50:37 +01:00

Implemented in 6eb30a43

Implemented in 6eb30a43

mvglasow commented 2026年03月15日 23:22:23 +01:00

I tested on a longer road trip over the past few days and did not experience any further crashes. This issue seems fixed.

I tested on a longer road trip over the past few days and did not experience any further crashes. This issue seems fixed.