Evaluate retrieval with labeled queries

A plausible result list is not enough evidence that a change improved search. Keep a small set of queries with independently chosen relevant document IDs. Run both versions on the same corpus and queries, then compare named metrics at the same cutoff. The labels should describe a user’s information need; they must not be inferred from whichever documents the current ranker prefers.

We compute three metrics for one query. Precision@k measures relevant results per requested slot. Recall measures the fraction of the judged relevant set that was retrieved. Reciprocal rank measures how soon the first relevant result appeared. A single result list cannot turn reciprocal rank into mean reciprocal rank; the latter requires a defined collection of queries and an averaging policy.

Choose a short-output convention and keep it visible. Here precision@4 for one returned relevant result is 1/4, because three requested slots are unfilled. Other reporting systems may choose a different convention; comparisons are meaningful only when both sides agree. Recall with an empty judgment set is defined as zero in this lab, rather than treating absence of labels as evidence of success.

ranked = ['x', 'b', 'a']
relevant = {'a', 'b', 'c'}
top = ranked[:3]
hits = sum(doc_id in relevant for doc_id in top)
print(round(hits / 3, 3), round(hits / len(relevant), 3))

All three metrics use the same cutoff. If a relevant item appears at position two while k is one, reciprocal rank is zero for this report. Repeated ranked IDs are invalid: returning one relevant document several times must not manufacture several hits. Repeated relevance judgments, however, refer to one relevant identity and collapse into a set.

Judgments have limits. An unjudged document is treated as nonrelevant by this small fixture, but in a large collection that may mean it was never assessed. A few labeled queries establish behavior on those queries, not universal retrieval quality. Keep those cases and labels separate from the implementation so a ranking change cannot quietly rewrite the exam.

ranked = ['x', 'a']
relevant = {'a'}
for k in (1, 2):
    first = next((i for i, value in enumerate(ranked[:k], 1) if value in relevant), None)
    print(k, 1 / first if first else 0.0)

Use the build to test fewer-than-k results, no results, no labels, duplicate IDs and a relevant item just beyond the cutoff. In the capstone, preserve the corpus, queries, judgments and scoring convention together. That small reproducible evaluation is a useful starting point for the larger semantic and hybrid retrieval experiments in the optional continuation.