List comprehension scope error from Python debugger

Question

In debugging my code, I want to use a list comprehension. However, it seems I cannot evaluate a list comprehension from the debugger when I'm inside a function.

I am using Python 3.4.

Script contents:

$ cat test.py 
#!/usr/bin/python

def foo():
    x = [1, 2, 3, 3, 4]

    print(x)

foo()

Interactive debugging:

$ python3 -mpdb test.py                                                                                                                                           
> /tmp/test.py(3)<module>()
-> def foo():
(Pdb) step
> /tmp/test.py(8)<module>()
-> foo()
(Pdb) 
--Call--
> /tmp/test.py(3)foo()
-> def foo():
(Pdb) 
> /tmp/test.py(4)foo()
-> x = [1, 2, 3, 3, 4]
(Pdb) 
> /tmp/test.py(6)foo()
-> print(x)
(Pdb) p [x for _ in range(1)]
*** NameError: name 'x' is not defined
(Pdb) p x
[1, 2, 3, 3, 4]

Why is x unknown to the list comprehension? How could I evaluate a list comprehension from the debugger, or achieve an equivalent behaviour? Is this a bug, or is it some sort of fundamental limitation to the debugger?

Answer 1

In Python 3, you have to use the interact command in pdb before you can access any non-global variables due to a change in the way comprehensions are implemented.

>>> def foo(): [][0]
... 
>>> foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in foo
IndexError: list index out of range
>>> import pdb;pdb.pm()
> <stdin>(1)foo()
(Pdb) x = 4
(Pdb) [x for _ in range(2)]
*** NameError: name 'x' is not defined
(Pdb) interact
*interactive*
>>> [x for _ in range(2)]
[4, 4]
>>> 

Answer 2

pdb seems to be running the code with:

eval(compiled_code, globals(), locals())

(or maybe even just eval(string, globals(), locals())).

Unfortunately, on compilation Python doesn't know of the local variables. This doesn't matter normally:

import dis

dis.dis(compile("x", "", "eval"))
#>>>   1           0 LOAD_NAME                0 (x)
#>>>               3 RETURN_VALUE

but when another scope is introduced, such as with a list comprehension of lambda, this compiles badly:

dis.dis(compile("(lambda: x)()", "", "eval"))
#>>>   1           0 LOAD_CONST               0 (<code object <lambda> at 0x7fac20708d20, file "", line 1>)
#>>>               3 LOAD_CONST               1 ('<lambda>')
#>>>               6 MAKE_FUNCTION            0
#>>>               9 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
#>>>              12 RETURN_VALUE

# The code of the internal lambda
dis.dis(compile("(lambda: x)()", "", "eval").co_consts[0])
#>>>   1           0 LOAD_GLOBAL              0 (x)
#>>>               3 RETURN_VALUE

Note how that's a LOAD_GLOBAL where x is in the local scope.

---

Here's a totally stupid hack to get around it:

(Pdb) eval("(lambda: x)()", vars())
[1, 2, 3, 3, 4]

Answer 3

Running List Comprehensions that depend on Outer Context in the Debugger

The accepted solution did not work for me. Or, maybe my setting differs in a way I cannot distinguish from the question's setting. But, this is where google led me when I searched for a solution to running list comprehensions that depend on objects outside of the comprehension's scope. So, here's a solution for other folks with the same problem. Run the following line in the debugger.

globals().update(locals())

Then, your list comprehension should run without any problems. Source.