One token policy for documents and queries

A query can match a document only when both reach the same representation. If indexing keeps CAFE while queries produce cafe, a dictionary lookup cannot know that you intended a caseless match. Define one tokenizer and call it on both paths. This is a retrieval policy, not a universally correct definition of a word.

Our policy normalizes with Unicode NFKC, applies casefold, then extracts alphanumeric runs excluding underscore. It keeps accents, numbers, order and repeated terms. Hyphens, apostrophes and underscores split runs. In particular, can't becomes can and t. That is a deliberate limitation to explain and test, not a claim about English morphology.

Case folding is stronger than lowercasing. It can change the number of code points, which matters when later work wants offsets into the original text. This course does not reuse normalized-string offsets as source offsets. Compatibility normalization also changes some presentation forms, such as fullwidth Latin characters and circled digits.

import unicodedata
print('Straße'.casefold())
print(unicodedata.normalize('NFKC', 'ＣＡＦＥ ①').casefold())

Keep multiplicity in document tokens. Converting red red blue into a set is useful for a membership question but loses the count needed for term frequency. Later, the ranking lesson deliberately deduplicates query terms so repeating a query word does not boost it. Those are two different contracts at two different boundaries.

Do not add stemming, stop-word removal or accent stripping accidentally. Each changes which strings can match and needs a corpus-specific evaluation. For now café and cafe remain different terms; an empty string has no terms.

import re
from collections import Counter
tokens = re.findall(r'[^\W_]+', 'red_red-blue')
print(tokens)
print(dict(Counter(tokens)))

Test the boundaries you promise: fullwidth forms, case-fold expansion, composed accents, punctuation, empty text and repeated terms. Record those choices beside your fixture corpus. If a later retrieval experiment changes the tokenizer, rebuild the index and reevaluate the same labeled queries; comparing a new query policy against an old index is not a fair ranking experiment.