Use GDB for Debugging Running Python Processes (Blog Post)

I did write lots of python code and had to debug many times. Usually, I use pdb — as probably most of us do. However, at some point, a running python process hung up and I wanted to figure out why without restarting the process in a debugger. This is one of the situations where pdb cannot help you anymore.

Well, first of all, I can run a strace on the process that is for sure. However, I need to get a Traceback and see where in the code the program is stuck. — Yes, indeed gdb is the tool to use and I actually used it a lot for debugging C code. Plus, I heard it also supports other than C programs, especially python programs. There is an official page manual page that describes how gdb can be used to debug a python program: https://wiki.python.org/moin/DebuggingWithGdb

Looks easy. You install gdb and python3.7-dbg using apt-get. Then you are ready to go. However, I observed that this is not the case if the program you want to debug is not executed by the system python but with another python interpreter, e.g. an interpreter of a conda environment. In this case, you probably face the problem I describe and solve below:

What did I do? Well first, I installed gdb and python3.7-dbg using apt-get:

apt-get install gdb python3.7-dbg

Now it is time to play. Let’s figure out the Process ID (PID) of the python process that hang up and attach gdb to the running process. This is simple. I used pgrep to get the PID of the program foo I executed earlier.

$ pgrep -f "python -m foo"
1234

Ok, the PID is 1234. Now let’s attach gdb as described in the official manual page:

$ gdb python 1234
...
Attaching to program: /usr/bin/python, process 1234
...
0x00007fb269180ff7 in ?? ()

Apparently, gdb successfully attached to the program /usr/bin/python and process 1234. However, I am worried it cannot dig into the program, because of the last line showing “??”. In fact, if I want to use py-bt to get a stack trace with python code, I get:

(gdb) py-bt
Unable to locate python frame
(gdb) cont

Using just bt doesn’t make it much better:

(gdb) bt
#0  0x00007fb269180ff7 in ?? ()
#1  0x000056397adff674 in ?? ()
#2  0x00007fb269803b90 in ?? ()
#3  0x000000003b9aca00 in ?? ()
#4  0x0000000000000000 in ?? ()

You probably already have an idea of what is wrong here. I executed the python module foo using a python interpreter of a conda environment. Yet, I am trying to attach gdb to the default python in /usr/bin/python and the process which is executed by another python interpreter. Let me correct myself. This should work out now:

$ gdb /path/to/cona/env/python 1234

This successfully attaches gdb to the process. I can now get a backtrace by typing bt. However, now gdb does not know py-bt anymore:

(gdb) py-bt
Undefined command: "py-bt".  Try "help".

As I found in this article (https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/), the problem is that the python binary of the conda environment has a different path. Thus, gdb fails to auto-load the python extensions. You can check this by typing:

(gdb) info auto-load

To enable the python extension, I need to add it manually:

source /usr/share/gdb/auto-load/usr/bin/python3.7-gdb.py

You can of course use .gdbinit to let gdb do this automatically during start. Thereto, create the file ~/.gdbinit and write the gdb command shown above in this file.

Now, gdb knows the command py-bt and it is possible to get the traceback:

(gdb) py-bt
Traceback (most recent call first):
  <built-in method sleep of module object at remote 0x7fb2696ae530>
  File "test.py", line 5, in <module>

Enjoy gdb with its python extension also while using not the system python interpreter but custom interpreters of virtual/conda environments!