Build one small tool whose design is easy to reuse and inspect. The reporter turns nested text documents into ranked word counts. It combines a recursive token source, a configurable pure transform, filtering, local aggregation and deterministic final ordering. The same configured reporter can process multiple independent document collections without retaining their counts.
The input model is deliberately limited: each document is a string leaf or an acyclic list/tuple tree of string leaves with bounded depth. The outer documents input is a finite iterable and may be one-shot. This is local data transformation; file reading, HTML generation, API calls and publishing belong to later application boundaries.
Separate token boundaries from normalization. iter_tokens splits each string leaf on whitespace and lazily yields the raw tokens in depth-first order. The default normalize_token strips outer whitespace, removes only the stated ASCII punctuation from both ends, and applies Unicode casefold. Interior punctuation survives: "don't" and "ab-c" remain single tokens. This precise policy is useful and testable without claiming linguistic completeness.
punctuation = ".,!?;:\"'()[]{}"
def normalize(token):
return token.strip().strip(punctuation).casefold()
raw = "Oak, OAK! ... ab-c".split()
print(tuple(normalize(token) for token in raw))Normalization precedes length filtering and counting. Two raw spellings can become one key, and punctuation-only tokens can become empty. min_length is at least one, so empty normalized results are excluded. A custom deterministic string transform is allowed; the exact same transform must prepare stop words at construction and data tokens during reporting. Comparing differently normalized stop words would make a seemingly obvious filter fail.
The factory captures a frozenset of transformed stop words and the minimum length. Snapshot the finite stop iterable once while constructing the reporter. A caller may later mutate its original list, and a one-shot stop generator may already be exhausted. Neither event should change the configured reporter. The supplied transform is expected to be deterministic and input-preserving; Python cannot enforce that expectation for arbitrary callables.
Source consumption is explicit. Equal document values supplied afresh produce the same report. Passing an exhausted document iterator again produces an empty report; advancing that iterator is an observable positional effect even when its elements remain unchanged.
Each invocation creates a new local count dictionary. Recursively supplied raw tokens are transformed once, filtered and accumulated in one traversal. Finally return tuple pairs ordered by decreasing frequency, resolving equal counts by ascending word. A key of (-count, word) expresses that contract directly. Sorting only by frequency lets insertion history influence ties and makes reports harder to compare.
counts = {"oak": 2, "elm": 2, "birch": 1}
ranked = tuple(sorted(counts.items(), key=lambda pair: (-pair[1], pair[0])))
print(ranked)Lazy token traversal avoids constructing a complete token list, but the final report still needs one count per distinct accepted word and a materialized vocabulary sort. It cannot finish a final ranking over a genuinely infinite document stream. The bounded recursive model also does not promise arbitrary depth or cycle handling.
Errors remain explicit. Reject invalid minimum lengths immediately. Raise ValueError when recursive iteration reaches an unsupported leaf, and propagate callback failures instead of returning a misleading partial report. Later input leaves may be untouched after failure; no rollback of callback effects is promised.
Use the checks to predict snapshot and sort behavior, the bug hunts to find reversed ranking and retained count state, and the fill to implement a fresh invocation. The full build must support custom policies, one-shot inputs, Unicode case folding, exact length boundaries and repeated independent calls. Once it passes, move the pipeline into a real application boundary with its own file, output and error contracts.