A cleaning pipeline has two questions to answer: what records did the source
contain, and which of those records satisfy the business rules? Keep the
questions separate. A parser establishes boundaries and representation. A later
schema assigns meaning. Turning every comma into a split point does neither
reliably: a quoted company name can contain a comma, and a quoted comment can
span physical lines. Python's csv module knows those boundaries.
CSV normally gives you text cells. An empty cell is "", and 001 remains the
text "001"; neither is automatically a number or a missing value. A JSON array
can carry numbers, booleans, nulls, strings and nested values. Loading JSON does
not prove the result is an array of records. It could be a scalar or one object.
Do not make a format's successful parsing stand in for your application's schema.
A parser preserves distinctions
import csv
import io
import json
text = 'id,name,visits\n001,"Lee, Ada",0\n'
row = next(csv.DictReader(io.StringIO(text, newline="")))
print(row['id'], row['name'], type(row['visits']).__name__)
print(json.loads('[{"visits":0,"note":null}]')[0])The first line prints a string-typed visit field. The JSON number becomes an
integer and null becomes None. Neither representation tells you whether zero
is plausible, whether a name is required, or whether an identifier is unique.
Those decisions belong in a documented schema.
Be careful before constructing dictionaries. dict(zip(header, row)) truncates
to the shorter input, so a ragged row can silently lose a field. Duplicate header
names overwrite earlier cells. Validate unique nonempty stripped header names
and exact width first. The build rejects a blank data record as wrong-width;
it does not quietly skip an observation. A trailing newline is a terminator,
not a new blank record.
Ambiguity must be caught before it disappears
Python's default JSON decoder keeps the final value for a repeated object member.
A dictionary cannot show that overwrite afterward. Use object_pairs_hook to
reject repeated names before constructing the object. Use parse_constant to
reject the non-standard NaN and infinity tokens accepted by the default decoder.
These checks apply inside nested objects too.
In the build, structural failure raises ValueError for the complete document.
Later lessons will keep valid rows while reporting bad business values; that is
a different boundary. Failing the document here prevents a caller from reporting
a partial import as complete. The local fixtures demonstrate parsing behavior,
not safe ingestion of arbitrarily large or remotely supplied documents.
Before practising, explain why splitting on newline is insufficient, predict the runtime types of CSV and JSON zeros, then construct one duplicate header and one nested duplicate JSON member. Both should be rejected without returning a table.