State a question before choosing a method. A baseline tells whether extra complexity helps; an ablation removes one component while keeping other settings fixed. Match data, budgets, seeds and scoring rules so differences have an interpretable source. A better score alone is not an explanation.
Freeze the method after development selection and before looking at the final held-out outcomes.
baseline = {"a": 0.0, "b": 1.0, "c": 0.0}
treatment = {"a": 1.0, "b": 1.0, "c": 0.0}
differences = [treatment[k] - baseline[k] for k in baseline]
print(sum(differences) / len(differences))Compare matched IDs rather than assuming arrays arrived in the same order. Reject missing or duplicate IDs and conflicting groups. Otherwise sorting or retries can silently pair unrelated cases and distort the effect estimate. Report support and effects per group, including a relevant distribution shift.
Choose decision thresholds with development data, then freeze them for held-out evaluation. Our helper selects the highest dev accuracy and breaks ties with the smallest candidate threshold. Specify ties rather than letting candidate input order decide. Do not repeatedly optimize on the held-out result.
y = [0, 1, 1]
p = [0.1, 0.4, 0.9]
for threshold in [0.3, 0.5]:
correct = sum((value >= threshold) == label for label, value in zip(y, p))
print(threshold, correct / len(y))A short research note should record the hypothesis, split unit, baseline, intervention, metric and failure policy before results. Afterwards add uncertainty, negative results, failure examples, limitations and the exact reproduction command. Keep exploratory observations distinct from the final held-out comparison. Avoid turning a toy result into a claim about real LLMs.
Join conditions by ID before subtracting. The same rows can arrive in a different order without changing the comparison.
The build enforces matched examples and dev threshold selection. Continue with Projects → Offline Python evaluation lab: it supplies a runnable synthetic baseline, a spurious-feature ablation and independent tests. Write your own hypothesis, change one factor, preserve a held-out split, and report what the evidence actually supports. Project checklist ticks are personal evidence records; only graded items contribute to mastery.