Need more information when errors occur · bloomberg/pystack · Discussion #254

aplund
Sep 11, 2025

I frequently get the error: "Invalid address in remote process", but don't know what to do about it. With verbose logging, there isn't actually an error reported. Could the nature of the error be logged so that one can figure out what's going wrong?

$ pystack remote $(pidof python) -v
INFO(process_remote): Analyzing process with pid 12105 using stack method StackMethod.AUTO with native mode NativeReportingMode.OFF
INFO(process_remote): Trying to stop thread 12105
INFO(process_remote): Waiting for thread 12105 to be stopped
INFO(process_remote): Thread 12105 stopped
INFO(process_remote): Trying to stop thread 12106
INFO(process_remote): Waiting for thread 12106 to be stopped
INFO(process_remote): Thread 12106 stopped
INFO(process_remote): Trying to stop thread 12107
INFO(process_remote): Waiting for thread 12107 to be stopped
INFO(process_remote): Thread 12107 stopped
INFO(process_remote): Trying to stop thread 12108
INFO(process_remote): Waiting for thread 12108 to be stopped
INFO(process_remote): Thread 12108 stopped
INFO(process_remote): Trying to stop thread 12109
INFO(process_remote): Waiting for thread 12109 to be stopped
INFO(process_remote): Thread 12109 stopped
INFO(process_remote): Trying to stop thread 12110
INFO(process_remote): Waiting for thread 12110 to be stopped
INFO(process_remote): Thread 12110 stopped
INFO(process_remote): Trying to stop thread 12111
INFO(process_remote): Waiting for thread 12111 to be stopped
INFO(process_remote): Thread 12111 stopped
INFO(process_remote): Trying to stop thread 12123
INFO(process_remote): Waiting for thread 12123 to be stopped
INFO(process_remote): Thread 12123 stopped
INFO(process_remote): Trying to stop thread 12124
INFO(process_remote): Waiting for thread 12124 to be stopped
INFO(process_remote): Thread 12124 stopped
INFO(process_remote): Trying to stop thread 12125
INFO(process_remote): Waiting for thread 12125 to be stopped
INFO(process_remote): Thread 12125 stopped
INFO(process_remote): All 10 threads stopped
INFO(parse_maps_file_for_binary): python binary first map found: VirtualMap(start=0x00000000003ff000, end=0x0000000000400000, filesize=0x1000, offset=0x0, device='103:02', flags='rw-p', inode=2322982, path='/home/austin.lund/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/bin/python3.13')
INFO(parse_maps_file_for_binary): Process does not have a libpython.so, reading from binary
INFO(parse_maps_file_for_binary): Heap map found: VirtualMap(start=0x000000004695c000, end=0x000000004697c000, filesize=0x20000, offset=0x0, device='00:00', flags='rw-p', inode=0, path='[heap]')
INFO(_get_bss): Determined exact addr of .bss section: 0x159cf40 (0x3ff000 + 0x119df40)
INFO(parse_maps_file_for_binary): bss map found: VirtualMap(start=0x000000000159cf40, end=0x0000000001620f68, filesize=0x84028, offset=0x119cf40, device='', flags='', inode=0, path='None')
INFO(process_remote): Attempting to find symbol '_PyRuntime' in /home/austin.lund/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/bin/python3.13
INFO(process_remote): Symbol '_PyRuntime' found at address 0x1557c40
INFO(process_remote): Attempting to find symbol '_PyRuntime' in /home/austin.lund/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/bin/python3.13
INFO(process_remote): Symbol '_PyRuntime' found at address 0x76dbe24fdc40
INFO(process_remote): Attempting to find symbol 'Py_Version' in /home/austin.lund/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/bin/python3.13
INFO(process_remote): Symbol 'Py_Version' found at address 0xf91d88
INFO(process_remote): Attempting to find symbol 'Py_Version' in /home/austin.lund/.local/share/uv/python/cpython-3.13.7-linux-x86_64-gnu/bin/python3.13
INFO(process_remote): Symbol 'Py_Version' found at address 0x76dbe1f37d88
INFO(get_python_version_for_process): Version found by scanning the bss section: 3.13
INFO(process_remote): Detaching from thread 12125
INFO(process_remote): Detaching from thread 12124
INFO(process_remote): Detaching from thread 12111
INFO(process_remote): Detaching from thread 12123
INFO(process_remote): Detaching from thread 12110
INFO(process_remote): Detaching from thread 12109
INFO(process_remote): Detaching from thread 12108
INFO(process_remote): Detaching from thread 12107
INFO(process_remote): Detaching from thread 12106
INFO(process_remote): Detaching from thread 12105
💀 Engine error: Invalid address in remote process 💀

Replies: 9 comments 6 replies

pablogsal
Sep 11, 2025
Maintainer

Interesting! This normally happens when we are failing to copy some memory from the process so it normally means either some of the internal structures are changing as we sample or that we have a bug somewhere .

Do you have a reproducer or a simple script we could run to try to get the error? Without that is a bit difficult to know why we're failing

0 replies

aplund
Sep 11, 2025
Author

I tried to narrow down what's going on. There's some randomness in it, but it seems that it happens when importing the "numba" package. I'll see if I can be more specific. But what's making it hard, is that when using pdb to debug the import process, sometimes it works!

0 replies

aplund
Sep 11, 2025
Author

Even more specifically, it seems to be the import of "llvmlite.binding". There's a bunch of other imports when you do that. It doesn't seem to be "llvmlite.ir", but again, there's a huge number of imports that happen and it's hard to figure out.

I have numba 0.61.0 and llvmlite 0.44.0. I could try some other versions and see if anything changes.

1 reply

@pablogsal

pablogsal Sep 11, 2025
Maintainer

Ah, this is interesting. numba is JIT compiling the process so a lot of those frames will be very difficult to analyze indeed. Ok if you have some reproducer it will be amazing, if not, I can try to give it a go on my side. Otherwise we can't ry a bunch of things. If you want meanwhile you can try to get a stack trace with a debugger. For example with gdb run pystack as:

$ gdb --args python -m pystack remote --native ...

and then do catch throw and run the process. Then try to get a backtrace on all the places where the breakpoint triggers and copy them to a file.

aplund
Sep 11, 2025
Author

The lines which trigger the problem aren't in numba. They are in llvmlite.binding.executionengine:

# XXX The ctypes function wrappers are created at the top-level, otherwise
# there are issues when creating CFUNCTYPEs in child processes on CentOS 5
# 32 bits.
_notify_c_hook = _ObjectCacheNotifyFunc(
 ExecutionEngine._raw_object_cache_notify)
_getbuffer_c_hook = _ObjectCacheGetBufferFunc(
 ExecutionEngine._raw_object_cache_getbuffer)

Comment them out and the tracing works. I don't know what it breaks.

1 reply

@pablogsal

pablogsal Sep 11, 2025
Maintainer

Oh, maybe what happens is that we are trying to read some python objects that we think we can read but they are altered by llvm? Without a reproducer or a stack trace is very difficult to know what are we failing to read :(

aplund
Sep 11, 2025
Author

I'm not sure if this is what you are suggesting, re stack trace:

Catchpoint 1 (exception thrown), 0x00007ffff70ae4a1 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0 0x00007ffff70ae4a1 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00007ffff7450092 in (anonymous namespace)::DirectoryReader::DirectoryReader (path="/proc/13822/task", this=<synthetic pointer>)
 at /opt/rh/devtoolset-10/root/usr/include/c++/10/ext/new_allocator.h:89
#2 pystack::getProcessTids (pid=<optimized out>) at src/pystack/_pystack/process.cpp:125
#3 0x00007ffff74b8d2e in pystack::ProcessTracer::ProcessTracer (this=this@entry=0x20f7e60, pid=pid@entry=13822) at src/pystack/_pystack/process.cpp:144
#4 0x00007ffff7481267 in __gnu_cxx::new_allocator<pystack::ProcessTracer>::construct<pystack::ProcessTracer, int&> (__p=0x20f7e60, this=<optimized out>)
 at /opt/rh/devtoolset-10/root/usr/include/c++/10/new:175
#5 std::allocator_traits<std::allocator<pystack::ProcessTracer> >::construct<pystack::ProcessTracer, int&> (__p=0x20f7e60, __a=...)
 at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/alloc_traits.h:512
#6 std::_Sp_counted_ptr_inplace<pystack::ProcessTracer, std::allocator<pystack::ProcessTracer>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<int&> (__a=...,
 this=0x20f7e50) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:551
#7 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<pystack::ProcessTracer, std::allocator<pystack::ProcessTracer>, int&> (__a=..., __p=<optimized out>,
 this=<optimized out>) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:682
#8 std::__shared_ptr<pystack::ProcessTracer, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<pystack::ProcessTracer>, int&> (__tag=..., this=<optimized out>)
 at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1371
#9 std::shared_ptr<pystack::ProcessTracer>::shared_ptr<std::allocator<pystack::ProcessTracer>, int&> (__tag=..., this=<optimized out>)
 at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:408
#10 std::allocate_shared<pystack::ProcessTracer, std::allocator<pystack::ProcessTracer>, int&> (__a=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:860
#11 std::make_shared<pystack::ProcessTracer, int&> () at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:876
#12 __pyx_pf_7pystack_8_pystack_14ProcessManager_2create_from_pid (__pyx_v_cls=0x7ffff754b500 <__pyx_type_7pystack_8_pystack_ProcessManager>, __pyx_v_pid=13822,
 __pyx_v_stop_process=<optimized out>) at src/pystack/_pystack.cpp:9911
#13 0x00007ffff74829e8 in __pyx_pw_7pystack_8_pystack_14ProcessManager_3create_from_pid (__pyx_v_cls=0x7ffff754b500 <__pyx_type_7pystack_8_pystack_ProcessManager>,
 __pyx_args=<optimized out>, __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>) at src/pystack/_pystack.cpp:9857
#14 0x0000000001b3dac9 in method_vectorcall.llvm[cold] ()
#15 0x00000000018a4842 in _PyObject_VectorcallTstate.llvm.2731793169297419064 ()
#16 0x00000000019b506b in PyObject_VectorcallMethod ()
#17 0x00007ffff746dee0 in __pyx_gb_7pystack_8_pystack_11generator2 (__pyx_generator=0x7ffff6bcf760, __pyx_tstate=0x159cdd0 <_PyRuntime+283024>, __pyx_sent_value=<optimized out>)
 at src/pystack/_pystack.cpp:16152
#18 0x00007ffff745c589 in __Pyx_Coroutine_SendEx (self=self@entry=0x7ffff6bcf760, value=0x1507668 <_Py_NoneStruct>, result=result@entry=0x7fffffffda30, closing=0)
 at src/pystack/_pystack.cpp:27713
#19 0x00007ffff7460c44 in __Pyx_Generator_Next (self=0x7ffff6bcf760) at src/pystack/_pystack.cpp:27979
#20 0x000000000181f62b in _PyEval_EvalFrameDefault ()
#21 0x00000000018bf494 in PyEval_EvalCode ()
#22 0x00000000018bfd86 in builtin_exec ()
#23 0x0000000001803dbf in cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.1191992843058394787 ()
#24 0x000000000182809d in _PyEval_EvalFrameDefault ()
#25 0x0000000001984237 in pymain_run_module ()
#26 0x0000000001983b38 in Py_RunMain ()
#27 0x00000000019838c5 in pymain_main.llvm ()
#28 0x00000000019836ad in main ()

Do I need to try and get more debugging symbols?

0 replies

aplund
Sep 11, 2025
Author

Sorry, that backtrace was junk. Wrong PID. This might be better:

#0 0x00007ffff70ae4a1 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00007ffff744ddb0 in pystack::ProcessMemoryManager::readChunkDirect (this=0x21e3670, addr=139270867684416, len=8, dst=0x7fffffffd0e8 "214円351円\n367円377円177円")
 at src/pystack/_pystack/mem.cpp:255
#2 0x00007ffff74b7b74 in pystack::AbstractProcessManager::copyObjectFromProcess<unsigned long> (destination=0x7fffffffd0e8, addr=139270867684416, this=0x21fb680)
 at src/pystack/_pystack/process.h:139
#3 pystack::AbstractProcessManager::setPythonVersionFromDebugOffsets (this=0x21fb680) at src/pystack/_pystack/process.cpp:700
#4 0x00007ffff7481884 in __pyx_pf_7pystack_8_pystack_14ProcessManager_2create_from_pid (__pyx_v_cls=0x7ffff754b500 <__pyx_type_7pystack_8_pystack_ProcessManager>,
 __pyx_v_pid=14369, __pyx_v_stop_process=<optimized out>) at src/pystack/_pystack.cpp:10066
#5 0x00007ffff74829e8 in __pyx_pw_7pystack_8_pystack_14ProcessManager_3create_from_pid (__pyx_v_cls=0x7ffff754b500 <__pyx_type_7pystack_8_pystack_ProcessManager>,
 __pyx_args=<optimized out>, __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>) at src/pystack/_pystack.cpp:9857
#6 0x0000000001b3dac9 in method_vectorcall.llvm[cold] ()
#7 0x00000000018a4842 in _PyObject_VectorcallTstate.llvm.2731793169297419064 ()
#8 0x00000000019b506b in PyObject_VectorcallMethod ()
#9 0x00007ffff746dee0 in __pyx_gb_7pystack_8_pystack_11generator2 (__pyx_generator=0x7ffff6bcf760, __pyx_tstate=0x159cdd0 <_PyRuntime+283024>, __pyx_sent_value=<optimized out>)
 at src/pystack/_pystack.cpp:16152
#10 0x00007ffff745c589 in __Pyx_Coroutine_SendEx (self=self@entry=0x7ffff6bcf760, value=0x1507668 <_Py_NoneStruct>, result=result@entry=0x7fffffffda30, closing=0)
 at src/pystack/_pystack.cpp:27713
#11 0x00007ffff7460c44 in __Pyx_Generator_Next (self=0x7ffff6bcf760) at src/pystack/_pystack.cpp:27979
#12 0x000000000181f62b in _PyEval_EvalFrameDefault ()
#13 0x00000000018bf494 in PyEval_EvalCode ()
#14 0x00000000018bfd86 in builtin_exec ()
#15 0x0000000001803dbf in cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.1191992843058394787 ()
#16 0x000000000182809d in _PyEval_EvalFrameDefault ()
#17 0x0000000001984237 in pymain_run_module ()
#18 0x0000000001983b38 in Py_RunMain ()
#19 0x00000000019838c5 in pymain_main.llvm ()
#20 0x00000000019836ad in main ()

0 replies

aplund
Sep 11, 2025
Author

Here's more with continuing:
traces.txt

0 replies

pablogsal
Sep 11, 2025
Maintainer

Yeah this is it! Looks like is this one:


Catchpoint 1 (exception thrown), 0x00007ffff70ae4a1 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0 0x00007ffff70ae4a1 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1 0x00007ffff744ddb0 in pystack::ProcessMemoryManager::readChunkDirect (this=0x21e3670, addr=139270867684416, len=656, dst=0x224e9e0 "") at src/pystack/_pystack/mem.cpp:255
#2 0x00007ffff74b7896 in pystack::Structure<pystack::py_runtime_v>::copyFromRemote (this=0x7fffffffd400)
 at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1324
#3 pystack::Structure<pystack::py_runtime_v>::copyFromRemote (this=0x7fffffffd400) at src/pystack/_pystack/structure.h:51
#4 pystack::Structure<pystack::py_runtime_v>::getField<pystack::FieldOffset<unsigned long> > (field=&pystack::py_runtime_v::o_finalizing, this=0x7fffffffd400)
 at src/pystack/_pystack/structure.h:80
#5 pystack::AbstractProcessManager::isInterpreterActive (this=<optimized out>) at src/pystack/_pystack/process.cpp:675
#6 0x00007ffff74658ed in __pyx_pf_7pystack_8_pystack_14ProcessManager_10interpreter_status (__pyx_v_self=<optimized out>) at src/pystack/_pystack.cpp:11249
#7 __pyx_pw_7pystack_8_pystack_14ProcessManager_11interpreter_status (__pyx_v_self=<optimized out>, __pyx_args=<optimized out>, __pyx_nargs=<optimized out>,
 __pyx_kwds=<optimized out>) at src/pystack/_pystack.cpp:11223
#8 0x00000000018a4842 in _PyObject_VectorcallTstate.llvm.2731793169297419064 ()
#9 0x00000000019b506b in PyObject_VectorcallMethod ()
#10 0x00007ffff7465bfb in __pyx_pf_7pystack_8_pystack_4_check_interpreter_shutdown (__pyx_self=<optimized out>, __pyx_v_manager=<optimized out>) at src/pystack/_pystack.cpp:14708
#11 __pyx_pw_7pystack_8_pystack_5_check_interpreter_shutdown (__pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>)
 at src/pystack/_pystack.cpp:14670
#12 0x00007ffff747daea in __pyx_gb_7pystack_8_pystack_8generator1 (__pyx_generator=0x7ffff6ac6170, __pyx_tstate=0x159cdd0 <_PyRuntime+283024>, __pyx_sent_value=<optimized out>)
 at src/pystack/_pystack.cpp:15302
#13 0x00007ffff745c589 in __Pyx_Coroutine_SendEx (self=self@entry=0x7ffff6ac6170, value=0x1507668 <_Py_NoneStruct>, result=result@entry=0x7fffffffd890, closing=0)
 at src/pystack/_pystack.cpp:27713
#14 0x00007ffff7460c44 in __Pyx_Generator_Next (self=0x7ffff6ac6170) at src/pystack/_pystack.cpp:27979
#15 0x00007ffff74609c0 in __Pyx_PyIter_Next_Plain (iterator=0x7ffff6ac6170) at src/pystack/_pystack.cpp:27309
#16 __Pyx_Generator_Yield_From (gen=gen@entry=0x7ffff6bcf760, source=<optimized out>, retval=retval@entry=0x7fffffffd928) at src/pystack/_pystack.cpp:28569
#17 0x00007ffff746e203 in __pyx_gb_7pystack_8_pystack_11generator2 (__pyx_generator=0x7ffff6bcf760, __pyx_tstate=0x159cdd0 <_PyRuntime+283024>, __pyx_sent_value=<optimized out>)
 at src/pystack/_pystack.cpp:16226
#18 0x00007ffff745c589 in __Pyx_Coroutine_SendEx (self=self@entry=0x7ffff6bcf760, value=0x1507668 <_Py_NoneStruct>, result=result@entry=0x7fffffffda30, closing=0)
 at src/pystack/_pystack.cpp:27713
#19 0x00007ffff7460c44 in __Pyx_Generator_Next (self=0x7ffff6bcf760) at src/pystack/_pystack.cpp:27979
#20 0x000000000181f62b in _PyEval_EvalFrameDefault ()
#21 0x00000000018bf494 in PyEval_EvalCode ()
#22 0x00000000018bfd86 in builtin_exec ()
#23 0x0000000001803dbf in cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.1191992843058394787 ()
#24 0x000000000182809d in _PyEval_EvalFrameDefault ()
#25 0x0000000001984237 in pymain_run_module ()
#26 0x0000000001983b38 in Py_RunMain ()
#27 0x00000000019838c5 in pymain_main.llvm ()
#28 0x00000000019836ad in main ()
(gdb) c
Continuing.
💀 Engine error: Invalid address in remote proce

that tells us we are failing to check if the interpreter is shutting down and we are failing to read some weird memory address when copying from the interpreter address. This looks suspicious because it's pointing to a very odd address. That looks weird because it seems that we resolved _PyRuntime correctly so we should have the offsets correctly found. Maybe we failed somehow. I need to play a bit bit a reproducer will surely help

0 replies

aplund
Sep 11, 2025
Author

I'm not sure how to pin this all down. I'm using a uv environment with python 3.13.7. I have llvmlite 0.44.0. I then just run python to get an interactive interpreter and do

>>> import llvmlite.binding

In another terminal I run pystack remote $(pidof -s python). I take it that's not enough information to reproduce?

4 replies

@pablogsal

pablogsal Sep 11, 2025
Maintainer

I can give it a go, will get back to you :)

@pablogsal

pablogsal Sep 12, 2025
Maintainer

I think I got it I will be investigating what's going on

@aplund

aplund Sep 12, 2025
Author

Thanks.

@pablogsal

pablogsal Sep 12, 2025
Maintainer

Found the problem will work on a fix soon. See #255

Uh oh!

Need more information when errors occur #254

Uh oh!

aplund Sep 11, 2025

Replies: 9 comments · 6 replies

Uh oh!

pablogsal Sep 11, 2025 Maintainer

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

pablogsal Sep 11, 2025 Maintainer

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

pablogsal Sep 11, 2025 Maintainer

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

pablogsal Sep 11, 2025 Maintainer

Uh oh!

aplund Sep 11, 2025 Author

Uh oh!

pablogsal Sep 11, 2025 Maintainer

Uh oh!

pablogsal Sep 12, 2025 Maintainer

Uh oh!

aplund Sep 12, 2025 Author

Uh oh!

pablogsal Sep 12, 2025 Maintainer

aplund
Sep 11, 2025

Replies: 9 comments 6 replies

pablogsal
Sep 11, 2025
Maintainer

aplund
Sep 11, 2025
Author

aplund
Sep 11, 2025
Author

pablogsal Sep 11, 2025
Maintainer

aplund
Sep 11, 2025
Author

pablogsal Sep 11, 2025
Maintainer

aplund
Sep 11, 2025
Author

aplund
Sep 11, 2025
Author

aplund
Sep 11, 2025
Author

pablogsal
Sep 11, 2025
Maintainer

aplund
Sep 11, 2025
Author

pablogsal Sep 11, 2025
Maintainer

pablogsal Sep 12, 2025
Maintainer

aplund Sep 12, 2025
Author

pablogsal Sep 12, 2025
Maintainer