A sensor that measured zero and a sensor that never reported are different
sources of evidence. Python truthiness groups many values together: zero, False,
None and empty text all test false. A cleaning policy must separate them.
value or 0 silently replaces absence with a measurement and also hides some
invalid inputs. A short expression can change the meaning of a whole report.
Define the vocabulary for this feed before aggregating it. The build treats an
absent value, None, blank text and the case-insensitive stripped tokens NA and
N/A as missing. It does not assume that every string meaning "unknown" in some
other dataset has that meaning here. The numeric string "0" is invalid under
this lesson's policy, whereas integer 0 is a valid observation. Schema conversion
and missingness classification are related but separate decisions.
The denominator is a policy too
values = [0, 8, None]
observed = [value for value in values if value is not None]
print(sum(observed), len(observed), sum(observed) / len(observed))
print(sum(observed) / len(values))The observed mean is 4.0. Dividing the same sum by all three rows gives a different answer, equivalent here to filling the missing observation with zero. Such imputation may occasionally be justified, but it must be disclosed rather than hidden inside a denominator. Excluding missing observations also does not prove that the observed subset represents the missing population.
The build reports input count, observed count, missing count, invalid positions, sum and mean. With no observed values, the sum is 0.0 and the mean is None. Zero is the identity for addition; it is not evidence that an empty sample had a zero average. Consumers should inspect support counts before displaying a statistic.
Finite input and finite output
A successful float conversion does not prove a usable number. NaN and both
infinities are float values. The build rejects them into the invalid-position
list. It also rejects booleans and integers too large to become finite floats.
Use math.isfinite after conversion. Check overflow explicitly instead of
allowing an infinite statistic to flow into a later report.
math.fsum reduces ordinary accumulation error; for example it preserves the
small contribution in [1e16, 1.0, -1e16]. It cannot guarantee that every sum
fits in a float. The build raises ValueError when otherwise finite observations
produce aggregate overflow. That distinguishes an unusable aggregate from an
ordinary row-level rejection.
Before practising, classify each of 0, False, None, "0", " na " and NaN.
Then verify count == observed + missing + len(invalid). Keep that equation
visible when changing the policy: a newly recognized missing token should move
a row between categories, never make it disappear.