traceback.walk_stack(None) does not behave the same as traceback.walk_stack(inspect.currentframe())

[VincentVanlaer]

Bug report

traceback.walk_stack(None) omits the calling frame, making it equivalent to traceback.walk_stack(inspect.currentframe().f_back). Given the documentation just says "If f is None, the current stack is used.", I would have expected traceback.walk_stack(None) to be equivalent to traceback.walk_stack(inspect.currentframe()).

Example code

The following code

import traceback
import inspect

def abc():
    print(list(traceback.walk_stack(None)))
    print(list(traceback.walk_stack(inspect.currentframe())))
    print(list(traceback.walk_stack(inspect.currentframe().f_back)))

abc()

has as output

[(<frame at 0x7f66ac559a40, file '/home/vincenttc/data/temp/test/test.py', line 10, code <module>>, 10)]
[(<frame at 0x7f66ac55fa60, file '/home/vincenttc/data/temp/test/test.py', line 7, code abc>, 7), (<frame at 0x7f66ac559a40, file '/home/vincenttc/data/temp/test/test.py', line 10, code <module>>, 10)]
[(<frame at 0x7f66ac559a40, file '/home/vincenttc/data/temp/test/test.py', line 10, code <module>>, 10)]

Note that the frame for abc is missing in the first and last line.

Your environment

• CPython versions tested on: 3.8.10 (Ubuntu 20.04), 3.10.5 (Arch Linux)

• PR: GH-96092: restore frame height of traceback.walk_stack to that of pyt… #99015

[iritkatriel]

On main your code doesn't work at all:

>>> import traceback
>>> import inspect
>>> 
>>> def abc():
...     print(list(traceback.walk_stack(None)))
...     print(list(traceback.walk_stack(inspect.currentframe())))
...     print(list(traceback.walk_stack(inspect.currentframe().f_back)))
... 
>>> abc()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in abc
  File "/Users/iritkatriel/src/cpython/Lib/traceback.py", line 333, in walk_stack
    f = sys._getframe().f_back.f_back.f_back.f_back
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'f_back'

I see that this function changed in

commit 5644c7b3ffd49bed58dc095be6e6148e0bb4431e
Author: Ammar Askar <ammar@ammaraskar.com>
Date:   Sun Jul 4 19:14:33 2021 -0400

    bpo-43950: Print columns in tracebacks (PEP 657) (GH-26958)
    
    The traceback.c and traceback.py mechanisms now utilize the newly added code.co_positions and PyCode_Addr2Location
    to print carets on the specific expressions involved in a traceback.
    
    Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>
    Co-authored-by: Ammar Askar <ammar@ammaraskar.com>
    Co-authored-by: Batuhan Taskaya <batuhanosmantaskaya@gmail.com>

This part:

@@ -309,7 +321,7 @@ def walk_stack(f):
     current stack is used. Usually used with StackSummary.extract.
     """
     if f is None:
-        f = sys._getframe().f_back.f_back
+        f = sys._getframe().f_back.f_back.f_back.f_back
     while f is not None:
         yield f, f.f_lineno
         f = f.f_back

CC @ammaraskar @pablogsal @isidentical

[pablogsal]

I am currently in release hell, @ammaraskar @isidentical can you take a look?

[ammaraskar]

Added details on the root cause here: #99015 (comment)

It looks like traceback.walk_stack was never really meant to be called without immediately using StackSummary.extract on it. It is a generator so the:

    if f is None:
        f = sys._getframe().f_back.f_back

is a bit misleading, those lines don't run when you call traceback.walk_stack but only when you start iterating over the generator.

So if you do something like:

def outer():
    return inner()
def inner():
    return traceback.walk_stack(None)

frames = outer()


def print_frames(f):
    print(list(f))

print_frames(frames)

You might expect it to print module -> outer -> inner but it would actually get module -> print_frames because that is where the walk_stack generator is consumed.

---

My intuition is we should fix walk_stack to return a generator that iterates over the frames from when it is called, rather than iterated over. While this is a breaking change, it seems like walk_stack(None) is almost never called in the stdlib or tests without an immediate StackSummary.extract nor did I find anything in a quick search over Github.

[graingert]

@ammaraskar we use it here: https://github.com/dask/distributed/blob/c137ac0856af75a7ee54259ff86a15f6be1babe4/distributed/client.py#L2929

[ammaraskar]

Thanks for the link. That use case is fine too since the generator is iterated over immediately. People tend to either pass into StackSummary.extract or iterate over it without storing the generator.