Generators & Lazy Evaluation
Stream data instead of materializing it. yield, generator pipelines, and why laziness matters when the data does not fit in memory.
Most Python code starts by building lists. That is fine until the list is a million log lines, a stream of model responses, or an infinite sequence. A generator lets you describe the next value without materializing every value.
Laziness is a control lever: lower memory, earlier results, and pipelines that can process data as it arrives. The cost is that a generator is one-shot. Once you consume it, it is gone.
The mental model
A generator function is a function that contains yield. Calling it does not run the body immediately. It returns a generator object. Each next() resumes the function until the next yield, pauses there, and remembers its local state.
def countdown(n):
while n > 0:
yield n
n -= 1
nums = countdown(3) # body has not run yet
next(nums) # 3
next(nums) # 2
list(nums) # [1]That pause/resume behavior is why generators can represent huge or infinite streams. They only hold the current frame and whatever local state the function needs.
Generator functions vs generator expressions
A generator expression is the lazy cousin of a list comprehension:
squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))The list computes and stores every square now. The generator expression computes one square each time something asks for it. Use generator expressions for simple map/filter shapes; use generator functions when the logic needs names, branches, try/finally, or multiple yield points.
def non_empty_lines(path):
with open(path) as f:
for line in f:
stripped = line.strip()
if stripped:
yield strippedThe file object is already lazy. This function keeps the laziness and adds a small transformation.
Lazy pipelines
Generators compose. Each stage consumes an upstream iterable and yields a downstream value. Nothing runs until the final consumer asks for data.
def strip_comments(lines):
for line in lines:
line = line.split("#", 1)[0].strip()
if line:
yield line
def parse_numbers(lines):
for line in lines:
yield int(line)
numbers = parse_numbers(strip_comments(open("values.txt")))
total = sum(numbers)This is the shape behind preprocessing pipelines: read, normalize, filter, chunk, embed. Each stage stays testable because it accepts any iterable, not just a file.
Eager [x*x for x in src]
Lazy (x*x for x in src)
itertools: the lazy toolbox
The itertools module gives you fast, memory-efficient building blocks:
from itertools import count, islice, pairwise
first_five = list(islice(count(start=10, step=2), 5))
# [10, 12, 14, 16, 18]
for before, after in pairwise([3, 8, 10]):
print(after - before)Reach for it before writing a custom loop for slicing, chaining, grouping, counting forever, or taking while a predicate is true. These tools are lazy by default, which keeps the pipeline streaming.
Exhaustion is real
A generator is not a reusable collection. It is a cursor over work that has not happened yet.
items = (n for n in range(3))
list(items) # [0, 1, 2]
list(items) # []If you need to iterate twice, either materialize once with list() or create a fresh generator each time. Treat this as a design decision, not an accident.
When to reach for this
Variations
Try it
These exercises are about preserving laziness. Mentally ask, “At this line, how many values are actually in memory?”
Read a file in chunks
Write read_chunks(path, size) so callers can process a large binary file a piece at a time. It should stop cleanly at EOF and close the file even if the caller breaks early.
Approach. Put the open inside the generator function and use a with block. Each yield hands one bytes chunk to the caller; when iteration ends, the context manager closes the file.
Show solution
def read_chunks(path: str, size: int = 1024 * 1024):
with open(path, "rb") as f:
while True:
chunk = f.read(size)
if not chunk:
return
yield chunk
for chunk in read_chunks("events.bin", size=4096):
process(chunk)This uses constant memory with respect to file size. The only bytes held by the reader are the current chunk. The with block still runs its cleanup when the generator is closed.
Build an infinite counter safely
Create a counter that yields start, then adds step forever. Show how to take only the first five values without hanging the program.
Approach. Infinite generators are fine when the consumer is bounded. Use a custom generator to see the mechanics, then itertools.islice to cap the read.
Show solution
from itertools import islice
def count_from(start: int = 0, step: int = 1):
current = start
while True:
yield current
current += step
first_five = list(islice(count_from(start=10, step=3), 5))
assert first_five == [10, 13, 16, 19, 22]The generator itself never ends. The safety comes from islice, which asks for exactly five values and then stops. Never call list() on an unbounded generator without a limiting wrapper.
Fix a consumed pipeline
You have a lazy pipeline of text lines. You need both the sum of parsed numbers and a preview of the first three parsed values. Why does the preview come out empty, and how should you structure it?
numbers = (int(line) for line in lines if line.strip())
total = sum(numbers)
preview = list(numbers)[:3] # []Approach. sum(numbers) consumes the generator. If you need two passes, make a factory that creates a fresh generator, or materialize once if the data is small enough.
Show solution
def parsed_numbers(lines):
for line in lines:
line = line.strip()
if line:
yield int(line)
def make_numbers():
return parsed_numbers(["10", "", "20", "30", "40"])
total = sum(make_numbers())
preview = []
for value in make_numbers():
preview.append(value)
if len(preview) == 3:
break
assert total == 100
assert preview == [10, 20, 30]A generator is a one-shot cursor. A generator function is reusable because it creates a new cursor each time. If the source is a file, reopen it or deliberately store the needed subset while streaming.