Deduplication begins with a question about identity: what does one row describe,
and which field identifies that entity? Equal names are usually poor business
keys; different customers can share a name. The build uses a case-sensitive text
id and strips only its boundary whitespace. It preserves leading zeros and
internal spaces because the contract provides no evidence that those distinctions
are accidental.
A duplicate key is not automatically a duplicate row. Two records for id A and name Ada may repeat the same evidence. Id A with names Ada and Grace expresses a conflict. A dictionary comprehension keyed by id silently keeps the final record in both situations. That produces convenient output while destroying the information needed to review the decision. Record the choice explicitly.
Compare the payload, not just the key
rows = [{'id': 'A', 'name': 'Ada'}, {'id': 'A', 'name': 'Grace'}]
print({row['id']: row['name'] for row in rows})
print(rows[0] == rows[1], rows[0]['id'] == rows[1]['id'])The comprehension yields only Grace. The equal key does not make the complete rows equal. In the build, compare normalized dictionaries using ordinary Python equality. A field that is absent differs from a field present with None. Extra payload fields participate in comparison. This is exact structural comparison, not fuzzy name matching or proof that either value is true.
The disclosed policy keeps the first usable row for each key and records all
later rows. An equal normalized row gets duplicate; a different row gets
conflict. Both issues include the source position and the first kept position.
A bad key gets a key issue with no first position. Keep-first is deterministic
for the same ordered input, but it is not order-independent or a source-quality
ranking.
Preserve provenance and ownership
Store the first row together with its original 1-based position. Otherwise the issue may point at an index in the filtered result rather than the source. Keep accepted rows in first-seen order and issues in source order. The reconciliation equation is input count equals kept count plus issue count, including invalid keys. An empty id should never enter the index and accidentally merge all missing-key rows.
The function makes shallow outer copies before replacing ids. That protects
source dictionaries from the function's normalization. It does not promise deep
isolation of nested payloads. State that limit rather than implying that dict
copy recursively clones a record. Deep copying is an independent choice with
its own costs and supported-value constraints.
Before the build, try "001" versus "1", "A" versus "a", and " A "
versus "A". Then construct a four-row sequence that alternates a kept payload
and a conflicting payload. Explain why the baseline must not change. A real
business process might quarantine all conflicting rows or prefer a trusted
source; it needs a different, equally auditable contract.