homepage

Trying to use any kind of python gdb integration results in the following error:
(gdb) py-bt
Traceback (most recent call first):
Python Exception <class 'gdb.error'> Invalid cast.: 
Error occurred in Python command: Invalid cast.
I have tracked it down to the _type_... globals, and I am able to fix it with the following commands:
(gdb) pi
>>> # Look up the gdb.Type for some standard types:
... _type_char_ptr = gdb.lookup_type('char').pointer() # char*
>>> _type_unsigned_char_ptr = gdb.lookup_type('unsigned char').pointer() # unsigned char*
>>> _type_void_ptr = gdb.lookup_type('void').pointer() # void*
>>> _type_unsigned_short_ptr = gdb.lookup_type('unsigned short').pointer()
>>> _type_unsigned_int_ptr = gdb.lookup_type('unsigned int').pointer()
After this, it works correctly. I was able to workaround it by making a fix_globals that resets the globals on each gdb.Command. I do not understand why the originally initialized types are not working properly. It feels like gdb-inception trying to debug python within a gdb that debugs cpython while executing python code.
I have tried this using hg/default cpython (--with-pydebug --without-pymalloc --with-valgrind --enable-shared) 
1) System install of gdb 7.11, linked against system libpython 3.5.1.
2) Custom install of gdb 7.11.50.20160411-git, the debug cpython I am trying to debug

I tried Python 3.6 with "./configure --with-pydebug --without-pymalloc --with-valgrind". The py-bt commands works as expect in gdb.
Can you try to modify python-gdb.py to get a traceback of your Invalid cast error?
> I am able to fix it with the following commands:
> (...) _type_char_ptr = gdb.lookup_type('char').pointer() # char*
These defines are already in Tools/gdb/libpython.py (python-gdb.py is just a copy). I don't understand why you have to redeclare these variables again.

I have done a bit more digging, turns out it is actually no problem at all to debug python in gdb with gdb with python support (at least using a fixed python-gdb-py).
Turns out the type->length of the the globally initialized ptr types is wrong: It is 4 instead of 8, causing the cast to fail. I suspect the initialization is done before the executable is loaded and gdb is using some default. To verify, I have put two prints in the global initialization and in a command invocation:
GOBAL INITIALIZATION: gdb.lookup_type('char').pointer().sizeof == 4
COMMAND INVOKE: gdb.lookup_type('char').pointer().sizeof == 8
I guess to be fully portable those types need to be looked up at least whenever gdb changes it's binary, but I have no idea if there is a hook for that. But it seems reasonable to just replace those globals with on-demand lookup functions or properties.
If you are interested in the actual python/c stack traces for the error:
Thread 1 "gdb" hit Breakpoint 1, value_cast (type=type@entry=0x2ef91e0, arg2=arg2@entry=0x32b13f0) at ../../gdb/valops.c:571
571	 error (_("Invalid cast."));
(gdb) py-bt
Traceback (most recent call first):
 <built-in method cast of gdb.Value object at remote 0x3260de0>
 File "[...]/python-gdb.py", line 1151, in proxyval
 field_str = field_str.cast(_type_unsigned_char_ptr)
 File "[...]/python-gdb.py", line 945, in print_traceback
 % (self.co_filename.proxyval(visited),
 File "[...]/python-gdb.py", line 1578, in print_traceback
 pyop.print_traceback()
 File "[...]/python-gdb.py", line 1761, in invoke
 frame.print_traceback()
(gdb) bt
#0 value_cast (type=type@entry=0x2ef91e0, arg2=arg2@entry=0x32b13f0) at ../../gdb/valops.c:571
#1 0x000000000052261f in valpy_do_cast (self=<gdb.Value at remote 0x3260de0>, args=<optimized out>, op=UNOP_CAST) at ../../gdb/python/py-value.c:525
#2 0x00007fc7ce2141de in PyCFunction_Call (func=func@entry=<built-in method cast of gdb.Value object at remote 0x3260de0>, args=args@entry=(<gdb.Type at remote 0x2ed0fb0>,), kwds=kwds@entry=0x0)
 at ../../Python-3.5.1/Objects/methodobject.c:109
#3 0x00007fc7ce2c8887 in call_function (pp_stack=pp_stack@entry=0x7ffffc81cec8, oparg=oparg@entry=1) at ../../Python-3.5.1/Python/ceval.c:4655
#4 0x00007fc7ce2c57ac in PyEval_EvalFrameEx (

> Turns out the type->length of the the globally initialized ptr types is wrong: It is 4 instead of 8, causing the cast to fail. I suspect the initialization is done before the executable is loaded and gdb is using some default.
Hum. I see two options:
* try late initialization: initialize types at the first usage
* remove variables: always lookup types at runtime
Would you like to try to implement these options?

The second option seems like the safest choice, attached is a patch that addresses just that.

New changeset e1c6f8895fd8 by Victor Stinner in branch '3.5':
python-gdb.py: get C types at runtime
https://hg.python.org/cpython/rev/e1c6f8895fd8
New changeset 6425728d8dc6 by Victor Stinner in branch 'default':
Merge 3.5: Issue #26799
https://hg.python.org/cpython/rev/6425728d8dc6 

New changeset e4561aad29e6 by Victor Stinner in branch '2.7':
Fix python-gdb.py: get C types on demand
https://hg.python.org/cpython/rev/e4561aad29e6
New changeset 952c89a10be6 by Victor Stinner in branch '3.5':
Issue #26799: Fix typo in Misc/NEWS
https://hg.python.org/cpython/rev/952c89a10be6 

> The second option seems like the safest choice, attached is a patch that addresses just that.
Thanks, I applied your patch (your a bugfix, you forgot one "return" ;-)).
I also inlined _type_void_ptr() into _sizeof_void_p(), since it was only called there.
Thanks for your bug report and your patch.
Can you please sign the PSF Contributor Agreement? https://www.python.org/psf/contrib/contrib-form/
It's required when you contribute to Python. In short, it warns you that Python is free software ;-)

Thank you for the quick integration and fixing the return. I have signed the electronic form yesterday.

Great!