Accuracy alone can hide failure on a rare class. For binary labels, true positives predict one on a one; false positives predict one on a zero; false negatives miss a one. Precision divides true positives by predicted positives; recall divides them by actual positives. State what happens when a denominator is zero rather than letting NaN spread through a report.
The same predictions produce four different kinds of counts. Keep their support visible before computing a rate.
labels = [1, 0, 1, 0]
probabilities = [0.9, 0.8, 0.4, 0.1]
predicted = [p >= 0.5 for p in probabilities]
tp = sum(y == 1 and p for y, p in zip(labels, predicted))
fp = sum(y == 0 and p for y, p in zip(labels, predicted))
print(tp, fp)A probability is more informative than a thresholded class. Brier loss is mean((probability-label)^2); lower is better. It scores confidence as well as correctness. Calibration asks whether events assigned probability p occur about p of the time. Here binary calibration bins positive-class probabilities and compares their average to the fraction of positive labels.
import numpy as np
p = np.array([0.1, 0.4, 0.8])
y = np.array([0, 1, 1])
print(round(float(np.mean((p - y) ** 2)), 6))Our expected calibration error (ECE) is the count-weighted absolute gap between average probability and positive frequency in fixed equal-width bins. Include p=1 in the final bin. Empty bins contribute zero; small samples and bin choices make ECE unstable, so inspect counts and reliability curves too. This is the positive-probability form, not the top-label confidence convention.
Calibration groups probabilities rather than just thresholded classes. The contribution of each bin is weighted by its example count.
Select thresholds and bin choices using development data, then lock them before test evaluation. The build reports raw counts as well as loss, and rejects malformed labels/probabilities rather than truncating mismatched arrays with zip. Test endpoints zero and one and probability exactly equal to a threshold.