Generators: how a Python function can pause in the middle and pick up where it left off

Most functions are like vending machines: you press a button, the whole thing runs, and one answer drops out. A generator is a function that can stop halfway, hand you a value, and wait. Later it carries on from the exact line where it stopped, with all its variables still in place.

Why would you want that? Because sometimes you need a million values but only look at them one at a time. Building all of them first wastes memory. A generator makes each one only when you ask.

The word that changes everything: yield

A normal function ends with return. When Python sees return, the function is finished and its local variables are thrown away. The keyword yield works differently: it hands a value out and pauses the function instead of ending it. Any function containing yield is a generator function.

def count_up(limit):
    print("  start")
    n = 1
    while n <= limit:
        print("  about to yield", n)
        yield n
        print("  resumed after", n)
        n += 1
    print("  finished")

g = count_up(2)
print("created, nothing printed yet")
print("next ->", next(g))
print("next ->", next(g))
try:
    next(g)
except StopIteration:
    print("StopIteration")

Running this with Python 3.13 prints:

created, nothing printed yet
  start
  about to yield 1
next -> 1
  resumed after 1
  about to yield 2
next -> 2
  resumed after 2
  finished
StopIteration

Look at the first line. Calling count_up(2) did not print "start". Calling a generator function runs none of its body. It gives you back a generator object (here stored in g), like a bookmark sitting before the first line. Only next(g) starts the work.

Try it: press next() and watch the pause

Notice what happens on the second press. The function does not start from the top. It resumes at print("resumed after", n), and n is still 1, because the pause kept the local variables alive. This is the whole trick: a generator is a function whose place and variables are saved between calls.

On the third press the loop ends, the function runs off its last line, and Python raises StopIteration. That is not a bug: it is the standard signal that says "no more values".

What a for loop does for you

You rarely call next() by hand. A for loop calls it repeatedly and quietly catches StopIteration to know when to stop:

def squares():
    for n in range(1, 4):
        yield n * n

for s in squares():
    print(s)
print(list(squares()))

1
4
9
[1, 4, 9]

Each call to squares() makes a fresh generator object, so the second use starts over from the beginning.

Why bother: memory

A list holds every value at once. A generator holds only its bookmark and variables. Compare a list of one million squares to the same thing written as a generator expression, which is a one-line generator written with round brackets:

import sys
big  = [n * n for n in range(1_000_000)]
lazy = (n * n for n in range(1_000_000))
print(sys.getsizeof(big))
print(sys.getsizeof(lazy))

8448728
200

The list takes about 8.4 million bytes for its own storage (the exact numbers vary by Python version and computer). The generator takes about 200 bytes, however many values it could produce. Nothing has been computed yet; lazy only knows how to compute the next one. A generator can even describe an endless sequence, such as doubling forever, because it never builds the whole thing:

def doubles():
    n = 1
    while True:
        yield n
        n *= 2

g = doubles()
print([next(g) for _ in range(5)])   # [1, 2, 4, 8, 16]

The catch: you can only go through it once

A generator remembers where it is, and it only moves forward. Once it is used up, it is empty, and it does not rewind:

lazy = (n * n for n in range(5))
print(list(lazy))
print(list(lazy))

[0, 1, 4, 9, 16]
[]

The second list(lazy) gives an empty list with no error, which surprises many beginners. If you need to go through the values twice, either make a new generator or store the values in a list.

So what, for your own code?

Reach for a generator when you process data one piece at a time and don't need all of it in memory: reading a big file line by line, producing numbers on demand, or building a pipeline where each step hands items to the next. If you need to count, index or loop over the values several times, a plain list is simpler and fine. Both are good tools; the generator's special power is waiting.