Reproducible artifacts & failure-aware reports

A useful result can be traced to its data, configuration and code. Canonical JSON plus a cryptographic hash gives a stable fingerprint for the supplied JSON-compatible contents. It is not a substitute for a code revision, package versions or a trustworthy data provenance record; log those separately too.

Fingerprints identify the supplied payload. Code revision and environment versions are separate provenance fields.

import hashlib, json


def fingerprint(value):
    data = json.dumps(
        value,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
        allow_nan=False,
    ).encode("utf-8")
    return hashlib.sha256(data).hexdigest()


print(fingerprint({"a": 1, "b": 2}) == fingerprint({"b": 2, "a": 1}))

Object key order does not change this hash; list order does. If dataset order matters to training, that distinction is useful. Reject nonfinite numbers rather than creating nonstandard JSON. Atomic writing of the final artifact belongs in a CLI: write a temporary file in the destination directory and replace the destination only after a successful full serialization.

A report must state its denominators. Here accuracy is correct/total expected attempts, coverage is successful/total, and success_accuracy is correct/successful or None if none succeeded. A success can still be incorrect; an error has no measured correctness and contributes zero to the total-denominator score.

total, successful, correct = 5, 4, 3
print(correct / total, successful / total, correct / successful)

Also report each group. A strong overall score can hide collapse on a small but relevant distribution shift. Count each ID once and reject duplicates, which might otherwise represent retried cases twice. This exercise uses an explicit conservative total-denominator policy; a different study can choose a different policy, but must document it and show failure counts.

Label every denominator explicitly; the three ratios below use the same five expected attempts but answer different questions.

The build combines a configuration/data fingerprint with stable group summaries. Test an all-error group, an incorrect successful response, Unicode identifiers, reordered object keys and an actual data/config change. No timestamps enter the fingerprint; run metadata can contain a timestamp separately.