Generators
A generator is a special kind of function that produces a sequence of values one at a time, instead of computing them all up front and handing back a finished collection. Any function that contains at least one yield statement is a generator function. Calling it does not run the function body immediately — it returns a generator object, and the body only starts executing when you ask for the next value (for example, by iterating over it or calling next() on it).
A First Generator Function
Here is a simple generator that yields three numbers. Notice that calling the function does not print anything or produce a list — it just creates a generator object.
def count_up_to_three():
yield 1
yield 2
yield 3
gen = count_up_to_three()
print(gen) # <generator object count_up_to_three at 0x...>
print(type(gen)) # <class 'generator'>
for number in gen:
print(number)
# 1
# 2
# 3count_up_to_three() returns a generator object, not the values 1, 2, 3. The function body only runs when something iterates over that generator — each iteration of the for loop resumes the function right where it left off, runs until the next yield, and hands back that value.
How Generators Differ From Regular Functions
A normal function runs from top to bottom every time you call it and returns exactly once (or raises). All of its local state disappears the moment it returns. A generator function behaves very differently:
Execution pauses and resumes — when a generator hits a
yield, it suspends execution and hands a value back to the caller, but it does not lose its local variables, its position in loops, or its call stack. The next call tonext()resumes exactly where it paused.Lazy evaluation — values are produced on demand, one at a time, rather than all being computed and stored before the caller sees anything. If the caller only ever asks for the first two values, only the code needed to produce those two values ever runs.
One-way, single-pass iteration — a generator does not support indexing or going backwards; it only knows how to produce "the next value" until it is exhausted, at which point it raises
StopIteration(handled automatically byforloops).
def demo():
print('starting')
yield 'a'
print('resumed after first yield')
yield 'b'
print('resumed after second yield')
gen = demo()
print(next(gen))
print('--- caller does other work here ---')
print(next(gen))Memory Efficiency: Before, With a List
A common place generators shine is when a computation would otherwise require building a huge list in memory before you can use it.
import sys
def squares_list(n):
result = []
for i in range(n):
result.append(i ** 2)
return result
big_list = squares_list(10_000_000)
print(sys.getsizeof(big_list)) # roughly 80,000,000+ bytes (~80 MB) just for the listAfter: The Same Logic As A Generator
Rewriting the function to yield each square instead of collecting them avoids building the collection at all — values are computed lazily, one at a time, as the caller asks for them.
import sys
def squares_generator(n):
for i in range(n):
yield i ** 2
gen = squares_generator(10_000_000)
print(sys.getsizeof(gen)) # roughly 100-200 bytes, regardless of n
total = 0
for square in gen:
total += square
print(total)The generator object itself stays tiny — a couple hundred bytes — no matter how large n is, because it only ever needs to remember where it currently is in the loop, not every value it has produced or will produce. The list version, by contrast, grows in proportion to n. This is the core trade-off: generators trade the ability to revisit or index past values for a dramatically smaller memory footprint.
Sending Values Into A Generator
Generators can also receive values from the caller, not just send values out. Calling .send(value) resumes the generator and makes value become the result of the yield expression that was paused. This is an advanced, less commonly needed feature — most generators only ever produce values — but it is worth knowing it exists.
def running_total():
total = 0
while True:
amount = yield total
total += amount
gen = running_total()
print(next(gen)) # 0 - advances to the first yield
print(gen.send(10)) # 10 - amount becomes 10, total becomes 10
print(gen.send(5)) # 15 - amount becomes 5, total becomes 15Delegating With yield from
When one generator wants to yield every value produced by another generator (or any iterable), yield from delegates to it directly, without needing a manual loop.
def inner():
yield 1
yield 2
yield 3
def outer():
yield 'start'
yield from inner()
yield 'end'
for value in outer():
print(value)
# start
# 1
# 2
# 3
# endyield from inner() is roughly equivalent to writing for value in inner(): yield value, but it is shorter, and it also correctly forwards .send() calls and return values between the delegating generator and the sub-generator.
Practical Example: Reading A Huge File Lazily
One of the most common real-world uses of generators is processing files that are too large to comfortably load into memory all at once — server logs, large CSV exports, and so on. A generator function can yield one line at a time, so at any given moment only a single line needs to be in memory.
def read_large_file(path):
with open(path, 'r', encoding='utf-8') as file:
for line in file:
yield line.rstrip('\n')
for line in read_large_file('server.log'):
if 'ERROR' in line:
print(line)Even if server.log is many gigabytes, read_large_file() never holds more than one line in memory at a time. Compare this to calling file.readlines(), which reads the entire file into a list of lines up front — for a large enough file that can exhaust available memory or simply slow the program down waiting on a giant read. The generator version starts producing lines (and letting the caller act on them) almost immediately, and its memory usage stays flat no matter how large the file grows.