CodeRigor

LESSON

Separate missing, invalid and zero values

Classify source amounts without collapsing malformed data, absence and legitimate zero into one value.

Missing, invalid and zero answer different questions. A blank amount may mean no value was supplied. “bad” is malformed. $0.00 is a legitimate value. A refund such as-$1.25 is negative but valid under the existing currency grammar.

A nullable integer column can represent known cents and pd.NA, but it cannot explain why a value is absent. Keep a separate status column. The final audit still rejects a missing required sales amount; this exercise shows how to retain the distinction before applying that business policy.

Separate missing, invalid and zero values
start / 3
Classify source amounts without collapsing malformed data, absence and legitimate zero into one value.
Separate missing, invalid and zero values
Classify source amounts without collapsing malformed data, absence and legitimate zero into one value.

The supplied currency parser uses a documented grammar and Decimal-to-integer conversion. The classifier accepts its exact cents only when representable in this lesson’s nullable Int64 schema. Out-of-range data is invalid here, not rounded.

The final report has a different, deliberate representation: its previous records policy allows arbitrarily large Python integer cents, so that capstone uses object storage for exact Python ints. A dtype is an application decision, not a rule that every monetary table must use the same representation.

Python
import pandas as pd
values=pd.Series(pd.array([0,None,29],dtype="Int64"))
print(values.isna().tolist())
print(values.eq(0).fillna(False).tolist())

errors=coerce can be useful only when the lost distinctions are accounted for elsewhere. Here the status column is the audit evidence: missing and invalid both have pd.NA cents, but their causes remain different. Zero is never used as a catch-all replacement.

The tests include nondefault and duplicate index labels. Constructing typed arrays against the copied source index preserves one result per source position instead of accidentally aligning a new Series by unrelated labels.

Keep the interpretation visible
start / 3
Column representation, accepted business policy and independent report checks are separate decisions.
Keep the interpretation visible
Column representation, accepted business policy and independent report checks are separate decisions.
Python
labels=["missing","invalid","valid"]
cents=[None,None,0]
print(list(zip(labels,cents)))
print("absence cause is a separate column")

Implement classification and prove that zero, missing and invalid remain distinct. Then carry that evidence into an explicit aggregation grain.

Actual Pandas 2.3.3 · Independent records comparison

Follow the same observations through each operation.

These tables were computed from the supplied fixture. Choose a step to inspect actual values and row counts. Full report parity includes accepted rows, rejection reasons, unmatched source positions and every summary field.

11source records
5accepted rows
6rejected rows
1unmatched accepted
Raw CSV · 11 rows

Eleven raw records retain text IDs, literal NA and empty amounts before business validation.

Raw CSV · actual fixture values; null is labelled missing and empty text stays distinct
sale_idcustomer_iddateamount
s0010012025-01-01$0.29
s0020022025-01-02$0
s003missing2025-01-03-$1.25
s0010012025-01-01$0.29
s0010012025-01-01$0.30
s004"" · empty text2025-01-04$4
s005NA2025-01-05$2
s0060012025-02-30$1
s0070012025-01-07$1,234.50
s0080012025-01-08bad
s0090012025-01-09"" · empty text
Accepted lesson-frame dtypessale_id · stringcustomer_id · stringdate · stringcents · Int64
Download the data, independent baseline, Pandas implementation and checks →

Watch & read

Free lectures, talks and readings that cover this skill from another angle. All resources →