AI Feature: Auto-debug loop + Regression test #7

tichmangono commented 2025年12月14日 15:22:29 +01:00

Auto-debug loop - you can even do an automated fix-test-iterate cycle with little human interaction, leveraging hot reload! I am most excited about this one, and that's the first idea that came to mind when I was watching your talk. So see my sketch below of the diagram of how you could do this. It would leverage AI and also provide context which you gather (look up Context Engineering for AI) , and takes advantage of the live/autoreload feature. So it uses almost all your core features here. After fixing bug using state, you can also automatically generate regression tests here!

Auto-debug loop - you can even do an automated fix-test-iterate cycle with little human interaction, leveraging hot reload! I am most excited about this one, and that's the first idea that came to mind when I was watching your talk. So see my sketch below of the diagram of how you could do this. It would leverage AI and also provide context which you gather (look up Context Engineering for AI) , and takes advantage of the live/autoreload feature. So it uses almost all your core features here. After fixing bug using state, you can also automatically generate regression tests here!

sczi commented 2025年12月15日 04:05:07 +01:00

No description provided.

sczi commented 2025年12月15日 04:15:01 +01:00

That is a good idea, it looks like someone has implemented something similar, integrating LLM into the python debugger for automatic root cause analysis and proposing fixes: https://github.com/plasma-umass/ChatDBG

And when developing inside containers or virtual machine's managed by incus, you can take a stateful snapshot (including the state of all running process), and snapshots are almost instant if the backing filesystem is btrfs or zfs. So eventually I want to support automatically snapshoting and spinning up a copy of the container, that an LLM agent can run wild in without risk to your system, to have agent loops for debugging or writing code or anything. For agents with more limited permissions where we just are worried about it messing up the process state, we could maybe just fork the process and then let it try out different fixes in the forked process.

That is a good idea, it looks like someone has implemented something similar, integrating LLM into the python debugger for automatic root cause analysis and proposing fixes: https://github.com/plasma-umass/ChatDBG And when developing inside containers or virtual machine's managed by incus, you can take a stateful snapshot (including the state of all running process), and snapshots are almost instant if the backing filesystem is btrfs or zfs. So eventually I want to support automatically snapshoting and spinning up a copy of the container, that an LLM agent can run wild in without risk to your system, to have agent loops for debugging or writing code or anything. For agents with more limited permissions where we just are worried about it messing up the process state, we could maybe just fork the process and then let it try out different fixes in the forked process.

tichmangono commented 2025年12月16日 01:30:51 +01:00

yes, i think https://github.com/plasma-umass/ChatDBG starts to go in this direction. I havent seen any current LLMs or tools that really fully do this, the distinction with what you are doing is that its not just a post-mortem after the crash with oneshot solution... its is live, reloading, can start before the crash... and the container snapshot would be cool on top of this could allow replay if say the AI made errors or hallucinates. Of course this will consume resources, so would require some clever thinking. I think this would be a truly novel idea if it pans out

yes, i think https://github.com/plasma-umass/ChatDBG starts to go in this direction. I havent seen any current LLMs or tools that really fully do this, the distinction with what you are doing is that its not just a post-mortem after the crash with oneshot solution... its is live, reloading, can start before the crash... and the container snapshot would be cool on top of this could allow replay if say the AI made errors or hallucinates. Of course this will consume resources, so would require some clever thinking. I think this would be a truly novel idea if it pans out

sczi commented 2026年05月28日 18:11:44 +02:00

I've got an initial version working now. Code is hot reloaded after LLM edits, and when you get an exception you can ask it to autodebug. It will start an OpenCode session in plan mode, and for initial context give it the string exception message and exception type, the source code of the top stack frame, local variables of the top stack frame, and a numbered list with function name and arguments for the rest of the call stack. And in addition to the standard opencode tools (grep, glob, read, bash, websearch, etc), it also has an execute_python tool, and in the python repl it has a frame_locals list that contains the local variables for all other stack frames if it wants to look at them, and a frame_source() function that will return the source code for a given stack frame if it wants to inspect other parts of the call stack. It will explain the root cause of the bug, plan a fix, and a regression test that reproduces the bug. Then if it looks good you switch to build mode and it will implement it, starting with the regression test and checking that the test fails initially, then implement the fix and check that the test now passes.

I fixed a couple bugs with it so far, maybe the most interesting was 6dab163, a crash I was getting when doing completions on some objects. It found the bug was actually in jedi, searched the web and linked the upstream issue, a lot faster than I could have figured that out myself.

So far I've just developed it based on 'vibes', seeing how it autodebugs exceptions I get and tweaking the prompt and setup. Eventually I want to test it out on some benchmarks of real world python bugs (JunoBench, BugsInPy, etc) to have some way of measuring with what prompt, context, tools, and repl environment it works best.

I've got an initial version working now. Code is hot reloaded after LLM edits, and when you get an exception you can ask it to autodebug. It will start an OpenCode session in plan mode, and for initial context give it the string exception message and exception type, the source code of the top stack frame, local variables of the top stack frame, and a numbered list with function name and arguments for the rest of the call stack. And in addition to the standard opencode tools (grep, glob, read, bash, websearch, etc), it also has an execute_python tool, and in the python repl it has a `frame_locals` list that contains the local variables for all other stack frames if it wants to look at them, and a `frame_source()` function that will return the source code for a given stack frame if it wants to inspect other parts of the call stack. It will explain the root cause of the bug, plan a fix, and a regression test that reproduces the bug. Then if it looks good you switch to build mode and it will implement it, starting with the regression test and checking that the test fails initially, then implement the fix and check that the test now passes. I fixed a couple bugs with it so far, maybe the most interesting was 6dab163, a crash I was getting when doing completions on some objects. It found the bug was actually in jedi, searched the web and linked the [upstream issue](https://github.com/davidhalter/jedi/issues/1990), a lot faster than I could have figured that out myself. So far I've just developed it based on 'vibes', seeing how it autodebugs exceptions I get and tweaking the prompt and setup. Eventually I want to test it out on some benchmarks of real world python bugs ([JunoBench](https://huggingface.co/datasets/PELAB-LiU/JunoBench), [BugsInPy](https://github.com/soarsmu/bugsinpy), etc) to have some way of measuring with what prompt, context, tools, and repl environment it works best.