CodeRigor

LESSON

Document identity and a reproducible corpus

Validate a small document corpus, keep opaque IDs separate from searchable text, and preserve a stable snapshot for evaluation.

A search result needs to point back to something stable. Its title may change and its text may match another note, but its document ID tells the rest of the system which record was indexed, ranked and judged. Keep those jobs separate. Search normalizes words; identity should follow the source’s explicit rules.

Our corpus is a supplied list of small records, not a walk over your home directory. Every record has exactly three string fields: id, title and text. Blank text is useful evidence: it is a real document with no indexed terms. Duplicate IDs are different: they make a result ambiguous, so the validator rejects the entire input.

Identity survives representation changes
start / 3
Two records may have equal words while retaining distinct source identities.
Identity survives representation changes
Two records may have equal words while retaining distinct source identities.

Validate before indexing. Start with the outer list, check each dictionary’s exact keys and string values, then check its ID. This order turns malformed input into the stated ValueError instead of an accidental KeyError or a partly updated shared structure. An ID with only whitespace is invalid; a nonblank ID is preserved exactly, including surrounding whitespace if supplied.

Python
ids = ['A', 'a', 'A']
print(len(ids), len(set(ids)))
print(' a ' == 'a')

Do not confuse a new output list with new records. Appending the original dictionary into another list still shares that dictionary. Our fields are all strings, so dict(record) is enough to make an independent record here. A future nested metadata schema would need a separate ownership decision; this shallow-copy argument would no longer cover nested lists.

A copy separates later edits
start / 3
The validated record and the original dictionary have equal values but different identity.
A copy separates later edits
The validated record and the original dictionary have equal values but different identity.
Python
record = {'id': 'notes-1', 'title': '', 'text': ''}
snapshot = dict(record)
snapshot['title'] = 'New title'
print(repr(record['title']), snapshot['id'])

In the build, test empty input, empty text, equal content under different IDs, duplicate IDs and an output edit. After a rejected corpus, check that the original records are unchanged. These tests establish a trustworthy input boundary for the next lessons. They do not establish file permissions or document access control; those are responsibilities of the application supplying this snapshot.

Watch & read

Free lectures, talks and readings that cover this skill from another angle. All resources →

Use the text representation and TF–IDF sections. The lab states its own exact tokenization and weighting conventions.