An evaluation starts with a question, a dataset and a scoring rule. Freeze the rule before inspecting final results. A model reply is untrusted data: parse it as data, never execute it with eval or treat its instructions as evaluator policy. Keep the expected answer separate from prompt text and model output.
Gold labels belong to the evaluator. The completion is parsed as data and cannot redefine the scoring rule.
import json
reply = '{"answer":"B"}'
row = json.loads(reply)
print(set(row) == {"answer"}, row["answer"] == "B")Exact-match grading is reproducible but narrow. It can miss semantic answers, and a model-judge rubric can introduce its own biases. Start with a small schema that has an independently checkable answer, then audit concrete failures. Here valid outputs contain exactly an answer key whose value is A/B/C/D or ABSTAIN. Extra prose, unknown choices and extra fields are errors.
expected = {"case-1": "A", "case-2": "B"}
replies = {"case-1": '{"answer":"A"}'}
for ident in expected:
print(ident, "present" if ident in replies else "missing")Include missing, malformed and abstaining responses in the total example count. A retry should not create a new example ID. Report coverage separately from successful-response accuracy so a system cannot improve its headline score simply by failing on difficult inputs.
For robustness studies, separate development attack templates from held-out families and include benign controls. A suspicious phrase is not itself evidence of a harmful output. Document what a case tests and avoid including the gold answer in the tested input. Our offline schema exercise establishes measurement mechanics; it does not measure a real model's safety.
A missing or malformed reply is still an expected example. Keep it in the support count.
The build grades every expected ID once and ignores unrelated reply IDs. Status distinguishes missing, invalid, abstain, correct and incorrect. Tests include instruction-like text to ensure it is treated as an invalid reply, rather than influencing the evaluator.