Iterators, generators & streaming batches

An iterable can produce an iterator; an iterator remembers its current position. next advances it and exhaustion raises StopIteration. A list can be iterated again, but a generator is ordinarily a single-pass stream.

A generator advances only when the consumer requests another result.

source = (x * x for x in range(3))
print(next(source))
print(list(source))
print(list(source))  # exhausted

A function containing yield creates a generator. Its body runs on demand, suspends at each yield, and continues on the next request. Returning a list first materializes everything; this can exhaust memory on a large dataset or hang on an infinite source. Do not make a separate counting pass over a stream.

def take_even(source):
    for x in source:
        if x % 2 == 0:
            yield x


it = take_even(iter([1, 2, 3, 4]))
print(next(it))
print(list(it))

A batching generator holds at most one chunk. After yielding a list, create a new list instead of clearing and reusing the old one: consumers may keep references to earlier chunks. The last partial chunk must survive exhaustion.

Rebinding the producer buffer keeps earlier yielded batches intact; clearing it mutates objects the consumer retained.

The build checks laziness with a source that records every requested element. Creating the batching iterator must not consume records; asking for the first chunk consumes exactly that chunk. Invalid size may raise when the generator is first advanced, because its body has not run before then. Empty sources yield no chunks. Test an infinite counter using only the first two batches.