CodeRigor

LESSON

Reconcile the records and DataFrame reports

Preserve the existing cleaning policy with real DataFrame deduplication/merging and compare complete audit results.

The capstone preserves the existing sales-v1 report: first valid customer/sale wins, later identical normalized rows are duplicates, conflicting payloads are conflicts, and valid sales with unknown customers remain accepted but unmatched.

The Pandas implementation shares scalar parsing/validation rules with the records version, then independently performs DataFrame deduplication, enrichment and totals. It never calls the records clean_sales function. Compare the entire report, not just a total or screenshot.

Reconcile the records and DataFrame reports
start / 3
Preserve the existing cleaning policy with real DataFrame deduplication/merging and compare complete audit results.
Reconcile the records and DataFrame reports
Preserve the existing cleaning policy with real DataFrame deduplication/merging and compare complete audit results.

The source loader keeps identifiers and literal NA text intact. Scalar validators normalize accepted fields. DataFrame duplicated/drop_duplicates chooses the first valid normalized row, while error records are sorted back into source order so invalid rows and duplicate findings remain comparable.

The joined cents column deliberately stores Python ints as object dtype. Its totals stay exact even beyond signed64-bit range. Public output converts missing customer names to None and scalar counts/cents to ordinary Python ints before JSON serialization.

Python
import pandas as pd
rows=pd.DataFrame({"id":["a","a","b"],"cents":[29,30,-125]})
print(rows.duplicated("id",keep="first").tolist())
print(rows.drop_duplicates("id",keep="first")["cents"].tolist())

The independent records baseline and DataFrame path must agree on accepted rows, both error lists, unmatched source positions and every summary field. The fixture includes zero, refunds, leading-zero IDs, literal NA text, duplicate/conflicting rows, invalid dates/amounts and unknown customers.

A separate large-integer case demonstrates exact policy preservation. It is reported to browser media as a decimal string rather than an unsafe JavaScript number. A parity result concerns these defined inputs and policy; it is not a claim that arbitrary inferred dtypes are interchangeable.

Keep the interpretation visible
start / 3
Column representation, accepted business policy and independent report checks are separate decisions.
Keep the interpretation visible
Column representation, accepted business policy and independent report checks are separate decisions.
Python
import pandas as pd
huge=99999999999999999999999999999999
values=pd.Series([huge,1],dtype=object)
print(values.sum()==huge+1)
print(type(values.iloc[0]).__name__)

Run the downloadable report comparison and public checks. Inspect the actual dtypes, mask, group and join views, then add an independent edge case without changing the baseline policy. The CLI uses only supplied local fixtures after dependency installation.

Cleanup removes only the extracted lab’s own virtual environment and output files you created. No account, production database or remote data source is used. Continue into broader analysis after complete report parity is established.

Download the complete lab. The archive includes runnable code, tests, fixtures and a README.

Actual Pandas 2.3.3 · Independent records comparison

Follow the same observations through each operation.

These tables were computed from the supplied fixture. Choose a step to inspect actual values and row counts. Full report parity includes accepted rows, rejection reasons, unmatched source positions and every summary field.

11source records
5accepted rows
6rejected rows
1unmatched accepted
Raw CSV · 11 rows

Eleven raw records retain text IDs, literal NA and empty amounts before business validation.

Raw CSV · actual fixture values; null is labelled missing and empty text stays distinct
sale_idcustomer_iddateamount
s0010012025-01-01$0.29
s0020022025-01-02$0
s003missing2025-01-03-$1.25
s0010012025-01-01$0.29
s0010012025-01-01$0.30
s004"" · empty text2025-01-04$4
s005NA2025-01-05$2
s0060012025-02-30$1
s0070012025-01-07$1,234.50
s0080012025-01-08bad
s0090012025-01-09"" · empty text
Accepted lesson-frame dtypessale_id · stringcustomer_id · stringdate · stringcents · Int64
Download the data, independent baseline, Pandas implementation and checks →

Watch & read

Free lectures, talks and readings that cover this skill from another angle. All resources →