Analogy

Imagine that you have created a remarkable machine that is capable of generating thousands and thousands of lightbulbs per day. The machine generates these lightbulbs in boxes with a unique serial number. You don't have enough space to store all of these lightbulbs at the same time (i.e., you cannot keep up due to storage limitation), so you would like to adjust it to generate lightbulbs on demand.

Python generators don't differ much from this concept. Imagine that you have a function called barcode_generator that generates unique serial numbers for the boxes. Obviously, you can have a huge number of such barcodes generated by the function. A wiser, and space efficient, option is to generate those serial numbers on-demand.

As you can see, we have a self-contained “function” to generate the next unique serial number each time. This function returns a generator! As you can see, we are not calling the function each time we need a new serial number, but we are using next() given the generator to obtain the next serial number.

Output:

How many lightbulbs to generate? 5
[10000, 10001, 10002, 10003, 10004]
Produce more? [Y/n]: y
How many lightbulbs to generate? 6
[10005, 10006, 10007, 10008, 10009, 10010]
Produce more? [Y/n]: y
How many lightbulbs to generate? 7
[10011, 10012, 10013, 10014, 10015, 10016, 10017]
Produce more? [Y/n]: n

To be more precise, this generator is a lazy iterator! These lazy iterators do not store their content in memory! The use of next in the previous example is the explicit way of obtaining the next item from the iterator. The implicit way is using for loops:

These lazy iterators are useful when you work with big sequences (for example, reading large files).

Imagine that you have created a remarkable machine that is capable of generating thousands and thousands of lightbulbs per day. The machine generates these lightbulbs in boxes with a unique serial number. You don't have enough space to store all of these lightbulbs at the same time, so you would like to adjust it to generate lightbulbs on-demand.

Python generators don't differ much from this concept. Imagine that you have a function called barcode_generator that generates unique serial numbers for the boxes. Obviously, you can have a huge number of such barcodes returned by the function, subject to the hardware (RAM) limitations. A wiser, and space efficient, option is to generate those serial numbers on-demand.

Note the next(barcode) bit.

As you can see, we have a self-contained “function” to generate the next unique serial number each time. This function returns a generator! As you can see, we are not calling the function each time we need a new serial number, but instead we are using next() given the generator to obtain the next serial number.

To be more precise, this generator is a lazy iterator! An iterator is an object that helps us traverse a sequence of objects. It's called lazy because it does not load all the items of the sequence in memory until they are needed. The use of next in the previous example is the explicit way to obtain the next item from the iterator. The implicit way is using for loops:

In other words, a generator looks like a function but behaves like an iterator.

Real-world application?

Finally, real-world applications? They are usually useful when you work with big sequences. Imagine reading a huge file from disk with billions of records. Reading the entire file in memory, before you can work with its content, will probably be infeasible (i.e., you will run out of memory).

An analogy could help to grasp the idea here:

Imagine that you have created a remarkable machine that is capable of generating thousands and thousands of lightbulbs per day. The machine generates these lightbulbs in boxes with a unique serial number. You don't have enough space to store all of these lightbulbs at the same time (i.e., you cannot keep up due to storage limitation), so you would like to adjust it to generate lightbulbs on demand.

Python generators don't differ much from this concept.

  Imagine that you have a function called barcode_generator that generates unique serial numbers for the boxes. Obviously, you can have a huge number of such barcodes generated by the function. A wiser, and space efficient, option is to generate those serial numbers on-demand.

Machine's code:

def barcode_generator():
    serial_number = 10000  # Initial barcode
    while True:
        yield serial_number
        serial_number += 1


barcode = barcode_generator()
while True:
    number_of_lightbulbs_to_generate = int(input("How many lightbulbs to generate? "))
    barcodes = [next(barcode) for _ in range(number_of_lightbulbs_to_generate)]
    print(barcodes)

    # function_to_create_the_next_batch_of_lightbulbs(barcodes)

    produce_more = input("Produce more? [Y/n]: ")
    if produce_more == "n":
        break

As you can see, we have a self-contained “function” to generate the next unique serial number each time. This function returns a generator! As you can see, we are not calling the function each time we need a new serial number, but we are using next() given the generator to obtain the next serial number.

Output:

How many lightbulbs to generate? 5
[10000, 10001, 10002, 10003, 10004]
Produce more? [Y/n]: y
How many lightbulbs to generate? 6
[10005, 10006, 10007, 10008, 10009, 10010]
Produce more? [Y/n]: y
How many lightbulbs to generate? 7
[10011, 10012, 10013, 10014, 10015, 10016, 10017]
Produce more? [Y/n]: n

Analogy

Imagine that you have created a remarkable machine that is capable of generating thousands and thousands of lightbulbs per day. The machine generates these lightbulbs in boxes with a unique serial number. You don't have enough space to store all of these lightbulbs at the same time (i.e., you cannot keep up due to storage limitation), so you would like to adjust it to generate lightbulbs on demand.

Python generators don't differ much from this concept. Imagine that you have a function called barcode_generator that generates unique serial numbers for the boxes. Obviously, you can have a huge number of such barcodes generated by the function. A wiser, and space efficient, option is to generate those serial numbers on-demand.

Machine's code:

def barcode_generator():
    serial_number = 10000  # Initial barcode
    while True:
        yield serial_number
        serial_number += 1


barcode = barcode_generator()
while True:
    number_of_lightbulbs_to_generate = int(input("How many lightbulbs to generate? "))
    barcodes = [next(barcode) for _ in range(number_of_lightbulbs_to_generate)]
    print(barcodes)

    # function_to_create_the_next_batch_of_lightbulbs(barcodes)

    produce_more = input("Produce more? [Y/n]: ")
    if produce_more == "n":
        break

As you can see, we have a self-contained “function” to generate the next unique serial number each time. This function returns a generator! As you can see, we are not calling the function each time we need a new serial number, but we are using next() given the generator to obtain the next serial number.

Output:

How many lightbulbs to generate? 5
[10000, 10001, 10002, 10003, 10004]
Produce more? [Y/n]: y
How many lightbulbs to generate? 6
[10005, 10006, 10007, 10008, 10009, 10010]
Produce more? [Y/n]: y
How many lightbulbs to generate? 7
[10011, 10012, 10013, 10014, 10015, 10016, 10017]
Produce more? [Y/n]: n

Lazy Iterators

To be more precise, this generator is a lazy iterator! These lazy iterators do not store their content in memory! The use of next in the previous example is the explicit way of obtaining the next item from the iterator. The implicit way is using for loops:

for barcode in barcode_generator():
    print(barcode)

This will print barcodes infinitely, yet you will not run out of memory.

These lazy iterators are useful when you work with big sequences (for example, reading large files).

An analogy could help to grasp the idea here:

Imagine that you have created a remarkable machine that is capable of generating thousands and thousands of lightbulbs per day. The machine generates these lightbulbs in boxes with a unique serial number. You don't have enough space to store all these lightbulbs at the same time (i.e., you cannot keep up due to storage limitation), so you would like to adjust it to generate lightbulbs on demand.

Python generators don't differ much from this concept.

Imagine that you have a function x that generates unique serial numbers for the boxes. Obviously, you can have a huge number of such barcodes generated by the function. A wiser, and space efficient, option is to generate those serial numbers on-demand.

Machine's code:

def barcode_generator():
    serial_number = 10000  # Initial barcode
    while True:
        yield serial_number
        serial_number += 1


barcode = barcode_generator()
while True:
    number_of_lightbulbs_to_generate = int(input("How many lightbulbs to generate? "))
    barcodes = [next(barcode) for _ in range(number_of_lightbulbs_to_generate)]
    print(barcodes)

    # function_to_create_the_next_batch_of_lightbulbs(barcodes)

    produce_more = input("Produce more? [Y/n]: ")
    if produce_more == "n":
        break

As you can see, we have a self-contained "function" to generate the next unique serial number each time. This function returns a generator! As you can see, we are not calling the function each time we need a new serial number, but we are using next() given the generator to obtain the next serial number.

Output:

How many lightbulbs to generate? 5
[10000, 10001, 10002, 10003, 10004]
Produce more? [Y/n]: y
How many lightbulbs to generate? 6
[10005, 10006, 10007, 10008, 10009, 10010]
Produce more? [Y/n]: y
How many lightbulbs to generate? 7
[10011, 10012, 10013, 10014, 10015, 10016, 10017]
Produce more? [Y/n]: n

An analogy could help to grasp the idea here:

Imagine that you have created a remarkable machine that is capable of generating thousands and thousands of lightbulbs per day. The machine generates these lightbulbs in boxes with a unique serial number. You don't have enough space to store all of these lightbulbs at the same time (i.e., you cannot keep up due to storage limitation), so you would like to adjust it to generate lightbulbs on demand.

Python generators don't differ much from this concept.

Imagine that you have a function called barcode_generator that generates unique serial numbers for the boxes. Obviously, you can have a huge number of such barcodes generated by the function. A wiser, and space efficient, option is to generate those serial numbers on-demand.

Machine's code:

def barcode_generator():
    serial_number = 10000  # Initial barcode
    while True:
        yield serial_number
        serial_number += 1


barcode = barcode_generator()
while True:
    number_of_lightbulbs_to_generate = int(input("How many lightbulbs to generate? "))
    barcodes = [next(barcode) for _ in range(number_of_lightbulbs_to_generate)]
    print(barcodes)

    # function_to_create_the_next_batch_of_lightbulbs(barcodes)

    produce_more = input("Produce more? [Y/n]: ")
    if produce_more == "n":
        break

As you can see, we have a self-contained “function” to generate the next unique serial number each time. This function returns a generator! As you can see, we are not calling the function each time we need a new serial number, but we are using next() given the generator to obtain the next serial number.

Output:

How many lightbulbs to generate? 5
[10000, 10001, 10002, 10003, 10004]
Produce more? [Y/n]: y
How many lightbulbs to generate? 6
[10005, 10006, 10007, 10008, 10009, 10010]
Produce more? [Y/n]: y
How many lightbulbs to generate? 7
[10011, 10012, 10013, 10014, 10015, 10016, 10017]
Produce more? [Y/n]: n