CodeRigor

FREE LESSON

Reliable job-state simulation

Integrate bounded admission, leases, retries, idempotent effects, ordering and restart snapshots in one explicit state machine.

A reliable job system must connect its policies. A capacity check that forgets running jobs, a retry that forgets the original deadline, or a restart that resets delivery tokens can break an otherwise plausible implementation. The capstone keeps the full job lifecycle in one inspectable state machine. Its name refers to modeling the state a durable system would need; the exercise itself remains in memory.

A new submission records stable identity, payload, admission order and deadline. The unresolved count includes ready, waiting and running jobs. Claim chooses due work, increments the total attempt count and allocates a new token. Completion is split into committing the modeled effect and acknowledging the current delivery. That split creates a precise place to simulate a lost acknowledgement.

The integrated job lifecycle
start / 3
Effect commit and delivery acknowledgement are separate transitions.
The integrated job lifecycle
Effect commit and delivery acknowledgement are separate transitions.

Suppose job a adds seven to an account, then the worker disappears before acknowledgement. Serialize a snapshot through JSON and construct a fresh JobStore. At the saved lease deadline, the old token expires. A new claim returns the same job ID with a higher attempt and a new token. Committing again finds the saved effect key and preserves balance seven; acknowledgement then finishes the new delivery.

Python
import json

state = {"next_token": 2, "balances": {"alice": 7}, "effects": {"a": ["alice", 7]}}
restored = json.loads(json.dumps(state))
if "a" not in restored["effects"]:
    restored["balances"]["alice"] += 7
print(restored["balances"]["alice"], restored["next_token"])

The complete build adds ownership, deadlines and policy transitions around that small deduplication example. Save configuration and the next-token counter along with jobs. Restoring only the ready list would forget effects, retry history and stale delivery identities. Snapshots are deeply copied: changing a caller's saved dict must not rewrite a running store, and restoring must not retain aliases to that dict.

Restart between commit and ack
start / 3
The snapshot carries effect identity and the next delivery token.
Restart between commit and ack
The snapshot carries effect identity and the next delivery token.

Ordering is scoped to an account. An earlier unresolved job blocks later jobs for that account even while waiting for backoff; another account can proceed. Terminal dead work unblocks the next job, so applications must decide whether skipping a failed predecessor is semantically acceptable. This contract chooses that behavior explicitly rather than hiding it behind a global FIFO claim.

Every timed operation first advances a monotonic fake clock. Job deadlines take precedence over lease expiry. Expired leases either requeue work or consume the last attempt into dead state. Transient failure schedules capped backoff only if the new start is before the original deadline; permanent failure stops immediately. Dead records retain the payload and reason, with no automatic replay in this capstone.

Use the fixtures as failure stories: commit/restart/redeliver, ack before commit, exact lease equality, last-attempt expiry, future same-account retry, conflict reuse and snapshot aliasing. A committed effect can remain even if completion later becomes dead. The model demonstrates reconstructible decisions and idempotent local effects; it does not prove atomic disk writes, concurrent transactions, exactly-once delivery, external effect fencing or production recovery. Applying these ideas to RabbitMQ requires the named language, service, broker and storage preparation in the optional continuation.

Watch & read

Free lectures, talks and readings that cover this skill from another angle. All resources →

Watch 2:00–8:30 and 20:30–33:50 to place the capstone’s producer and consumer roles beside a real broker client. Use the primary guide for recovery; local snapshots do not run RabbitMQ.