CodeRigor

LESSON

Load CSV without guessing identifiers or missing data

Preserve leading-zero IDs and literal NA text with deliberate CSV settings and a checked schema.

CSV does not tell you whether001 is an identifier, an integer or a string that must retain its leading zero. Nor does it tell you whether the literal text NA is a customer ID or a missing-value marker. The application owns that interpretation.

The loader preserves every raw field as a Pandas string and checks the existing four-column sales schema. Scalar validation happens afterward. This keeps loading separate from deciding whether a date or currency amount is valid business data.

Load CSV without guessing identifiers or missing data
start / 3
Preserve leading-zero IDs and literal NA text with deliberate CSV settings and a checked schema.
Load CSV without guessing identifiers or missing data
Preserve leading-zero IDs and literal NA text with deliberate CSV settings and a checked schema.

dtype=string prevents numeric inference. keep_default_na=False and na_filter=False prevent default missing-value token interpretation. An empty amount remains raw text until the validation policy decides what it means.

The standard-library CSV precheck retains the existing exact-width/header contract before Pandas loading. Source row counts parsed records. A quoted newline belongs to a single record, so it does not create a new source-row number. Header-only input still needs explicit column dtypes.

Python
import pandas as pd,io
frame=pd.read_csv(io.StringIO("id,name\n001,NA\n"),dtype="string",keep_default_na=False,na_filter=False)
print(frame.iloc[0].to_dict())

A parser can successfully load a row that the application should reject. A blank date or malformed currency stays visible for the later audit instead of disappearing during import. Literal NA/NULL strings also remain distinguishable from missing values.

Validate the header even when there are no data records. Reordered allowed headers are normalized to one canonical order. Ragged or duplicate-column input is a schema error, not a partially successful report.

Keep the interpretation visible
start / 3
Column representation, accepted business policy and independent report checks are separate decisions.
Keep the interpretation visible
Column representation, accepted business policy and independent report checks are separate decisions.
Python
import csv,io
text='id,note\n"a\nb",one\nc,two\n'
rows=list(csv.reader(io.StringIO(text,newline="")))
print("data records",len(rows)-1)
print(repr(rows[1][0]))

Implement deliberate loading and challenge literal missing-token text, reordered headers, empty tables and quoted fields. Next select and derive columns without accidental alignment.

Watch & read

Free lectures, talks and readings that cover this skill from another angle. All resources →