Paginate without loops or conflicting duplicates

A page can be valid while a collection is wrong. The next cursor can loop. Boundary records can overlap. A retry or concurrent update can return the same id with a different amount. Infinite requests and double-counted money are both bugs.

The collector uses a supplied fetch_page(cursor) function. That makes pagination testable without a socket. The desktop lab will connect it to request encoding, bounded reads and retry policy.

None has two roles in this endpoint protocol: it starts the first request, and a returned next None ends pagination. A loop that checks while cursor is not None before the first fetch would skip the entire API.

Before each fetch, check cancellation, cursor history and max_pages. Even a page with no items counts as work. Mark the cursor before calling the supplied fetcher so a repeated token cannot trigger another request.

pages = {None:{"ids":["a"],"next":"opaque:+0"}, "opaque:+0":{"ids":["a","b"],"next":None}}
cursor, calls = None, []
while True:
    calls.append(cursor)
    page = pages[cursor]
    cursor = page["next"]
    if cursor is None: break
print(calls)

Merge records by stable id. Equal repeated records are ignored, preserving first-seen order. Different records with the same id raise id-conflict. This lab chooses rejection rather than guessing which version is newer. A real API with version timestamps may support another explicit reconciliation policy.

max_items counts unique accepted records, so a harmless duplicate does not consume another slot. Check the limit before inserting a new id. Keep accepted dictionaries independent of the raw page.

Cancellation is cooperative. Check before fetching and again after a result arrives, before accepting it. The second guard prevents stale acceptance, but it does not make an already sent request disappear. Later concurrency lessons retain that distinction.

records = {}
for record in [{"id":"a","cents":125},{"id":"a","cents":125},{"id":"b","cents":200}]:
    old = records.get(record["id"])
    if old is not None and old != record:raise ValueError("conflict")
    if old is None:records[record["id"]] = dict(record)
print(list(records))
print(sum(r["cents"] for r in records.values()))

Complete merging and termination. Test an empty terminal page, duplicate overlap, a conflicting id, a repeated cursor, a page-budget stop and cancellation immediately after a fetch. Inspect fetch call counts as well as returned values.

Next we decide which individual failed requests may be retried, and how much deliberate waiting that policy may spend.