AI Feature: Trace-based debugging #5

tichmangono commented 2025年12月14日 15:21:32 +01:00

Trace-based debugging (you already talk about this) - capture the function calls with args and return values, then feed these to LLM to detect problems e.g. return values mismatch or unexpected behavior

Trace-based debugging (you already talk about this) - capture the function calls with args and return values, then feed these to LLM to detect problems e.g. return values mismatch or unexpected behavior

sczi commented 2025年12月15日 03:53:46 +01:00

What more can we expose to an LLM to help it debug? probably also all logging output, and some access to the values of everything in memory, like if it is a web application the request and response object will be important. Any ideas people have from real world experience with hard to debug issues is welcome, if you think for a bit about what information we could expose that would have helped tracked down that bug faster. I'd like everything presented to the user in a nice and quickly searchable way, but also available to LLMs, they can probably identify some bugs a lot faster than a human.

What more can we expose to an LLM to help it debug? probably also all logging output, and some access to the values of everything in memory, like if it is a web application the request and response object will be important. Any ideas people have from real world experience with hard to debug issues is welcome, if you think for a bit about what information we could expose that would have helped tracked down that bug faster. I'd like everything presented to the user in a nice and quickly searchable way, but also available to LLMs, they can probably identify some bugs a lot faster than a human.

tichmangono commented 2025年12月16日 01:41:09 +01:00

yes, req/res from website is a great example! perhaps database queries with params and results; tracking object mutations and what got modified and when; tracking timing of queries for example to avoid or trace race conditions and all these other edge cases that are often hard to find. Since this mode is likely to be deeper and more involved, maybe you can allow user to choose "verbose" when they are really hunting tricky bugs that often go unnoticed or fail silently. This way you can manage the view and user gets some control.

yes, req/res from website is a great example! perhaps database queries with params and results; tracking object mutations and what got modified and when; tracking timing of queries for example to avoid or trace race conditions and all these other edge cases that are often hard to find. Since this mode is likely to be deeper and more involved, maybe you can allow user to choose "verbose" when they are really hunting tricky bugs that often go unnoticed or fail silently. This way you can manage the view and user gets some control.