Program crashes with SIGSEGV when connecting to BLE while monitor rtt is enabled #2089

kat-brickxter commented 2025年03月05日 13:33:20 +01:00 (Migrated from github.com)

The program crashes on my nrf52840 when I connect to Bluetooth LE with "Program received signal SIGSEGV, Segmentation fault." It only crashes when I start the program from the blackmagic probe and when monitor rtt is enabled. When monitor rtt is disabled or I read the RTT messages with a JLink device, the program works without problems.

Image
Image

The program crashes on my nrf52840 when I connect to Bluetooth LE with "Program received signal SIGSEGV, Segmentation fault." It only crashes when I start the program from the blackmagic probe and when monitor rtt is enabled. When monitor rtt is disabled or I read the RTT messages with a JLink device, the program works without problems.  

dragonmux commented 2025年03月05日 13:40:05 +01:00

What version of BMD are you running (mon ver to check), and what are the steps to reproduce? If you're using the "SoftDevice" block from Nordic then this is known to have problems running under debug, especially if halting is involved in any way.

BMD v2.0, for now, has to halt the target to do RTT things as we don't know for certain if there aren't side-effects to accessing memory without halting (nobody has yet documented if it causes cycle stealing either from the core or from the bus matrix so we can lay out the limitations of doing this) and before blanket enabling this for this target (note: it cannot be done for all Cortex-M targets, not all support reading without halting! Eg, TM4C123 parts give garbage back if you don't halt first) we want to get these limitations documented so we can have some place to point to when people find it doesn't work quite how they thought it would (if it turns out there is cycle stealing going on).

What version of BMD are you running (`mon ver` to check), and what are the steps to reproduce? If you're using the "SoftDevice" block from Nordic then this is known to have problems running under debug, especially if halting is involved in any way. BMD v2.0, for now, has to halt the target to do RTT things as we don't know for certain if there aren't side-effects to accessing memory without halting (nobody has yet documented if it causes cycle stealing either from the core or from the bus matrix so we can lay out the limitations of doing this) and before blanket enabling this for this target (note: it cannot be done for all Cortex-M targets, not all support reading without halting! Eg, TM4C123 parts give garbage back if you don't halt first) we want to get these limitations documented so we can have some place to point to when people find it doesn't work quite how they thought it would (if it turns out there is cycle stealing going on).

kat-brickxter commented 2025年03月05日 13:53:44 +01:00 (Migrated from github.com)

I used 1.10.2, however I just updated to 2.0.0-rc2 and found the same behavior.
I use the nRF Connect SDK with the SoftDevice controller.

Steps:

1. I open gdb with arm-none-eabi-gdb -ex "target extended-remote /dev/ttyACM0" -ex "monitor swd_scan" -ex "attach 1"
   GNU gdb (GDB) 15.2
2. monitor rtt
3. r
4. program runs and RTT messages are shown in the minicom
5. I try to connect to the μC with my phone via BLE
6. The program crashes with SIGSEGV as shown above

Second scenario:

1. I open gdb with arm-none-eabi-gdb -ex "target extended-remote /dev/ttyACM0" -ex "monitor swd_scan" -ex "attach 1" GNU gdb (GDB) 15.2
2. monitor rtt disable
3. r
4. program runs
5. I connect to the μC with my phone and start receiving data without a crash

JLink:

1. The programm is already running on its own on the μC
2. I connect and open the RTT terminal in VSCode via the Nordic VS Code Plugin
3. I can read the RTT messages
4. I can connect with my phone via BLE and the program doesn't crash

It seems like the halting really is the issue as you describe it. It sounds like the JLink is able to access the data without halting as I can just connect it and open the RTT terminal without going into a debug mode or restarting the firmware.

I used 1.10.2, however I just updated to 2.0.0-rc2 and found the same behavior. I use the nRF Connect SDK with the SoftDevice controller. Steps: 1. I open gdb with arm-none-eabi-gdb -ex "target extended-remote /dev/ttyACM0" -ex "monitor swd_scan" -ex "attach 1" GNU gdb (GDB) 15.2 2. monitor rtt 3. r 4. program runs and RTT messages are shown in the minicom 5. I try to connect to the μC with my phone via BLE 6. The program crashes with SIGSEGV as shown above Second scenario: 1. I open gdb with arm-none-eabi-gdb -ex "target extended-remote /dev/ttyACM0" -ex "monitor swd_scan" -ex "attach 1" GNU gdb (GDB) 15.2 2. monitor rtt disable 3. r 4. program runs 5. I connect to the μC with my phone and start receiving data without a crash JLink: 1. The programm is already running on its own on the μC 2. I connect and open the RTT terminal in VSCode via the Nordic VS Code Plugin 3. I can read the RTT messages 4. I can connect with my phone via BLE and the program doesn't crash It seems like the halting really is the issue as you describe it. It sounds like the JLink is able to access the data without halting as I can just connect it and open the RTT terminal without going into a debug mode or restarting the firmware.

sidprice commented 2025年03月05日 16:27:26 +01:00 (Migrated from github.com)

@dragonmux @kat-brickxter I have seen a very similar behavior when debugging a Bluetooth application on nRF5340 using a JLink when a breakpoint is hit, i.e. the DUT is halted. In my opinion, the BT stack and/or the Soft Device just falls apart when the MCU is halted. To date I have not found any work-around for this.

I have seen it using CortexDebug in VSCode and also with Ozone, the Segger debugger.

@dragonmux @kat-brickxter I have seen a very similar behavior when debugging a Bluetooth application on nRF5340 using a JLink when a breakpoint is hit, i.e. the DUT is halted. In my opinion, the BT stack and/or the Soft Device just falls apart when the MCU is halted. To date I have not found any work-around for this. I have seen it using CortexDebug in VSCode and also with Ozone, the Segger debugger.

ALTracer commented 2025年03月05日 16:46:17 +01:00 (Migrated from github.com)

There is PR2037 from 2 months ago which closes this issue if you just need to continue working/debugging with BMP, please give that unmerged branch a try. Ping me if I should rebase it into recently tagged v2.0.0-rc2. However, I did not dedicate any time to documenting the amount of bus cycle stealing.

There is PR2037 from 2 months ago which closes this issue if you just need to continue working/debugging with BMP, please give that unmerged branch a try. Ping me if I should rebase it into recently tagged v2.0.0-rc2. However, I did not dedicate any time to documenting the amount of bus cycle stealing.

sidprice commented 2025年03月05日 17:33:38 +01:00 (Migrated from github.com)

@ALTracer Thanks, that may help the OP's issue.

@ALTracer Thanks, that may help the OP's issue.

kat-brickxter commented 2025年03月06日 10:04:49 +01:00 (Migrated from github.com)

@ALTracer Thank you! PR2037 fixed the issue for me. RTT and BLE now work at the same time. Merging the branch would be greatly appreciated.

@ALTracer Thank you! PR2037 fixed the issue for me. RTT and BLE now work at the same time. Merging the branch would be greatly appreciated.

dragonmux commented 2025年03月06日 11:47:00 +01:00

As mentioned on that PR too.. until we can document what the drawbacks are for hard real-time operations by having someone with one of the parts check for CPU freezes (detectable from SysTick halts while idle) or bus freezes (DMA halts, or, depending on access priority, CPU to peripheral access stutters) on access by the debugger, there are too many possible side-effects that #1532 made it extremely clear would be unacceptable for hard real-time usage. This is why this must happen first so there are no nasty surprises and we can have a table people can refer to that documents what happens on various parts with the TOPT enabled.

As mentioned on that PR too.. until we can document what the drawbacks are for hard real-time operations by having someone with one of the parts check for CPU freezes (detectable from SysTick halts while idle) or bus freezes (DMA halts, or, depending on access priority, CPU to peripheral access stutters) on access by the debugger, there are too many possible side-effects that #1532 made it extremely clear would be unacceptable for hard real-time usage. This is why this must happen first so there are no nasty surprises and we can have a table people can refer to that documents what happens on various parts with the TOPT enabled.

kat-brickxter commented 2025年03月06日 11:57:43 +01:00 (Migrated from github.com)

I understand that there are a lot of potential side-effects. However I think a flag which is clearly labeled as experimental within the configuration of native.ini would be a good solution until the full documentation is completed. I think that using RTT during time critical applications such as BLE isn't a complete edge-case and therefore this could be interesting for a lot of people at their own risk.

I understand that there are a lot of potential side-effects. However I think a flag which is clearly labeled as experimental within the configuration of native.ini would be a good solution until the full documentation is completed. I think that using RTT during time critical applications such as BLE isn't a complete edge-case and therefore this could be interesting for a lot of people at their own risk.