CodeRigor

FREE LESSON

Poison messages & controlled replay

Separate transient failures from permanent poison work, retain diagnostics and make replay an explicit decision.

A malformed message cannot become valid merely because a worker reads it again. A permission denial may require a configuration fix. A temporary dependency outage may be worth retrying. Treating every failure identically can leave a poison message cycling forever, consuming capacity while useful jobs wait.

This queue accepts an explicit processing outcome for each attempt: success, transient failure or permanent failure. The exercise does not pretend to infer the class from arbitrary exception text. The worker/application supplies the classification; the queue applies its stated policy. Count each attempt before deciding whether any budget remains.

One failed attempt, three outcomes
start / 3
A permanent error never spends another retry allowance.
One failed attempt, three outcomes
A permanent error never spends another retry allowance.

Transient failures with attempts left move to the ready tail. With a and b waiting, a failed attempt for a allows b to run next. That improves access for other work but changes completion order. Applications that need per-account effects in order must apply a narrower ordering rule; the capstone allows other accounts through while an earlier account job waits.

Python
from collections import deque

ready = deque([{"id": "a", "attempts": 0}, {"id": "b", "attempts": 0}])
job = ready.popleft()
job["attempts"] += 1
ready.append(job)
print([job["id"] for job in ready])

Permanent failure goes directly to quarantine, regardless of unused attempts. Transient failure also goes there once the total attempt allowance is consumed. A quarantine entry retains the ID, payload, attempt count and reason. These facts help a person distinguish a permanently invalid input from a job that repeatedly encountered a temporary failure. A dead state describes handling policy; it does not prove that an earlier attempt produced no partial external effect.

Explicit replay is a fresh budget
start / 2
Quarantine is inspected before a deliberate tail admission.
Explicit replay is a fresh budget
Quarantine is inspected before a deliberate tail admission.

A dead-letter list must not become a second infinite retry loop. Normal processing never automatically returns a dead job to ready. replay(job_id) is an explicit operator decision after a fix or investigation: remove that quarantine entry, clear the reason, reset the attempt count and append it at the ready tail. A second replay returns false because the entry is already absent. Resetting the budget is deliberate in this contract, not an implicit exception to the old limit.

This lesson assumes input IDs are unique and payloads are strings. The build validates the policy number and the outcome before removing any work. An invalid outcome must leave the next job untouched, even when the queue is empty. State views are deep copies so inspecting diagnostics cannot rewrite quarantine membership or reasons.

RabbitMQ's dead-letter exchange routes messages under configured broker conditions; it is not simply this Python list and does not magically repair poison data. Its documentation also describes safety and cycle behavior that depend on broker topology and queue type. Here the fixtures test classification, exact attempt limits, tail placement, retained diagnostics and deliberate replay without any broker. The next lesson integrates these policies with leases, a retry clock, ordered effects and restart snapshots while keeping the same honest in-memory boundary.

Watch & read

Free lectures, talks and readings that cover this skill from another angle. All resources →

Watch 20:30–33:50 to trace a consumer handling a delivery. Contrast receive/ack with the explicit failure outcomes here; quarantine and controlled replay are taught by the lesson and RabbitMQ guide.