9

Below is a generator function.

def f():
   x=1
   while 1:
      y = yield x
      x += y

Does this generator function (f) get implemented, internally, as shown below?

class f(collections.Iterable):
   def __init__(self):
      self.x = 1
   def __iter__(self):
      return iter(self)
   def __next__(self):
      return self.x
   def send(self, y):
      self.x += y
      return self.next()

Edit:

This is the answer for my question.

9
  • 1
    You can test whether they behave the same for yourself. Going into the internal implementation details seems too broad for a SO question.
    – jonrsharpe
    Commented Aug 16, 2017 at 22:25
  • 1
    This post may provide some info: aosabook.org/en/500L/a-web-crawler-with-asyncio-coroutines.html
    – Chris
    Commented Aug 16, 2017 at 22:27
  • 2
    of course they aren't equivalent, one is a class and one is a function
    – wim
    Commented Aug 16, 2017 at 22:29
  • 1
    Possible duplicate of What does the "yield" keyword do?
    – pppery
    Commented Aug 17, 2017 at 0:25
  • 2
    The short answer is that generators are not implemented internally as shown in your pure python class. Instead, they share most of the same logic as regular functions. Commented Aug 17, 2017 at 6:36

1 Answer 1

30

Internally, a generator works about the same as a regular function call. Under-the-hood, running generators and running functions use mostly the same machinery.

When you call either a function or a generator, a stackframe is created. It has the local variables (including the arguments passed into the function), a code pointer to the active opcode, and a stack for pending try-blocks, with-blocks, or loops.

In a regular function, execution begins immediately. When return is encountered, the final result is kept and the stackframe is freed along with everything it referenced.

In a generator function, the stackframe is wrapped in a generator-iterator object and returned immediately. The code in the generator function only runs when called by next(g) or g.send(v). Execution is suspended when yield is encountered.

One way to think of generators is that they are like functions that can be paused with yield and resumed with g.next(). The stackframe is kept alive, so resuming a running generator is much cheaper than making a new function call which has to build a new frame on every invocation.

4
  • 1
    How yield pause/suspend the function internally? Commented Aug 17, 2017 at 6:14
  • 3
    @overexchange Execution of both generators and functions involves running the current opcode and updating the code pointer. Pausing just means to stop doing that. Resuming means to continue doing that. Count with me, 1, 2, 3, and now talk about something else, and continue couting 4, 5, 6, ... All you need to know is the last count. Likewise, the stackframe keeps the state of the function and you can resume stop updating it and resume updating it at any time. Commented Aug 17, 2017 at 6:20
  • Building block for async is about task capable to suspend/resume. For server side programming, generally task is suspended on performing IO. When u say, pausing is to stop, are we not pausing because we are waiting for an IO(say http get request)? Commented Aug 17, 2017 at 6:25
  • 8
    @overexchange I think you're imagining this to be more complex than it is. The code for next(g) boils down to basically PyEval_EvalFrameEx(gen->gi_frame, exc). That's it, it just execs the current state of the frame (see Objects/genobject.c which is clear and simple in Python 2.7). A yield simply returns from that call. In contrast, a function call runs PyEval_EvalCodeEx(...) which creates a new frame and execs it as shown above. See Objects/funcobject.c. Commented Aug 17, 2017 at 6:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.