Async Job Results Arrive Out of Order: Merge by Key
Async job results come back in any order. Map each job ID to its URL with a keyed merge, rebuild the batch order, and flag missing or duplicate results.
On this page
Why Async Job Results Arrive Out of Order and What Breaks
Submit five scrapes against store.example.com in a tidy loop and the job IDs come back in order: J-101 through J-105. That ordering survives exactly as far as the submission response. After that, each job runs independently — one page is static HTML, another needs a headless browser, a third gets retried after a slow proxy hop. Completion order is a function of page weight and queue scheduling, not of when you submitted. Your webhook receiver hears about jobs whenever they finish.
Run a real batch once and the console tells the story:
submitted: ['J-101', 'J-102', 'J-103', 'J-104', 'J-105']
arrived: ['J-104', 'J-101', 'J-105', 'J-102', 'J-103']
Nothing failed. Every job completed. The damage happens downstream, in code that assumes arrival order equals submission order. The naive pipeline looks like this:
rows = list(zip(product_urls, results_as_arrived))
zip never throws. It pairs position 0 of your URL list with position 0 of whatever happened to land first, and the corruption is silent:
| Slot | URL from your input list | Result appended at that slot | What the row now claims |
|---|---|---|---|
| 0 | https://store.example.com/products/101 | J-104 payload | product 104’s price, filed under 101 |
| 1 | https://store.example.com/products/102 | J-101 payload | product 101’s price, filed under 102 |
| 2 | https://store.example.com/products/103 | J-105 payload | product 105’s price, filed under 103 |
| 3 | https://store.example.com/products/104 | J-102 payload | product 102’s price, filed under 104 |
| 4 | https://store.example.com/products/105 | J-103 payload | product 103’s price, filed under 105 |
Four out of five rows are wrong and nothing alerted you. If your current code does this and hasn’t broken yet, that’s luck, not a contract — the next batch with a heavier page in the middle scrambles differently.
The fix has three parts: a job-ID-to-URL map built at dispatch time, a merge that writes each result into a slot dictated by its job ID, and post-merge validation that flags missing and duplicate results instead of papering over them.
Building the Job-ID-to-URL Lookup Map Before Dispatch
Dispatch is the only moment you hold the job ID and the target URL in the same hand. Capture the pair then. Once the request is in flight, results arrive carrying only a job ID — if you never recorded what that ID was for, no amount of parsing recovers it after the fact.
The examples below target the FineData async endpoints, but the merge pattern itself is transport-agnostic; swap the base URL and everything still holds.
import asyncio
import httpx
API = "https://api.finedata.ai"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}
PRODUCT_URLS = [
f"https://store.example.com/products/{n}" for n in range(101, 106)
]
async def submit(client: httpx.AsyncClient, url: str) -> tuple[str, str]:
resp = await client.post(
f"{API}/api/v1/async/scrape",
headers=HEADERS,
json={"url": url, "formats": ["markdown"]},
)
resp.raise_for_status()
return url, resp.json()["job_id"]
async def build_job_map() -> dict[str, str]:
async with httpx.AsyncClient(timeout=30) as client:
pairs = await asyncio.gather(
*(submit(client, u) for u in PRODUCT_URLS)
)
jobs = dict(pairs) # job_id -> url
print(jobs)
return jobs
# {'J-101': 'https://store.example.com/products/101',
# 'J-102': 'https://store.example.com/products/102',
# 'J-103': 'https://store.example.com/products/103',
# 'J-104': 'https://store.example.com/products/104',
# 'J-105': 'https://store.example.com/products/105'}
Note the invariant, stated as a comment because it belongs in the code, not in a wiki page:
# Invariant: every dispatched job_id must exist in this map before
# any await on results begins. A result whose job_id is absent from
# the map is an orphan -- its URL is unrecoverable.
asyncio.gather returns submission responses in list order, which is fine — the scramble happens later, at completion. One variant worth calling out: if you use the batch endpoint (POST /api/v1/async/batch with a requests array and a callback_url), the same rule applies. Record each returned job ID next to its input URL at submission time. The batch webhook fires when all jobs complete, and the per-job result list is not a positional promise across a network boundary. For more on the jobs-and-webhooks model, see Async Scraping at Scale: Jobs, Batches, Webhooks.
Merging Results by Key: Filling a Pre-Sized Result Array
With the map in hand, the merge is short. Freeze the submission order, build an index from job ID to position, pre-size a result array with None, and write each incoming result into the slot its job ID dictates:
order = list(jobs) # submission order: J-101 .. J-105
idx_map = {job_id: pos for pos, job_id in enumerate(order)}
results: list = [None] * len(order)
for r in incoming: # arrival order: J-104, J-101, J-105, ...
results[idx_map[r.job_id]] = r
Printed side by side, before and after:
arrival order : ['J-104', 'J-101', 'J-105', 'J-102', 'J-103']
merged array : ['J-101', 'J-102', 'J-103', 'J-104', 'J-105']
Every write is a dict lookup plus a list assignment — both O(1). You are not sorting anything, and you are not searching for where a result belongs; the job ID tells you directly. The pre-filled None values do double duty: any slot still None after the merge is a job that never returned, which is exactly what the next section checks.
Flagging Missing Results: Detecting Jobs That Never Returned
After the merge, walk the array and report the holes. This is a ten-line check that most pipelines skip, and skipping it is how a 4,998-row report ships instead of a 5,000-row one with a note attached:
import logging
log = logging.getLogger("merge")
for pos, r in enumerate(results):
if r is None:
job_id = order[pos]
url = jobs[job_id]
log.warning("%s (%s) returned no result", job_id, url)
With job J-103 timed out and dropped, the log line reads:
WARNING:merge:J-103 (https://store.example.com/products/103) returned no result (slot 2 empty; gave up after 180s)
One caution before you re-scrape anything: an empty slot is not proof of loss. The job may still be running, or it may have failed server-side. Check the job status endpoint before spending money on a replacement:
async def job_status(client: httpx.AsyncClient, job_id: str) -> str:
resp = await client.get(
f"{API}/api/v1/async/jobs/{job_id}",
headers=HEADERS,
)
resp.raise_for_status()
return resp.json()["status"]
The status field is one of pending, processing, completed, failed, or cancelled — the authoritative set lives in the job status reference, and it’s worth checking against the live docs if you branch on it, because a status value you don’t recognize should fail loudly rather than silently fall through as “not terminal.”
Only re-drive the URL when the status is terminal and unsuccessful. Re-scraping a job that is still processing doubles your token spend for zero additional data. When you do re-drive, drive only the failures rather than the whole batch — Re-Drive Only the Failures: Handling Partial Batch Results covers that loop in detail.
Detecting Duplicate Results: The Same Job ID Returned Twice
Webhook delivery is usually at-least-once. Your receiver returns a 500 during a deploy, or takes too long to ACK, and the provider retries — same job, same payload, delivered twice. You can also manufacture duplicates yourself by polling GET /api/v1/async/batch/{batch_id} with include_results=true while a webhook for the same batch is in flight. Two delivery channels, one result, two arrivals.
The merge must flag a second write to an occupied key instead of silently overwriting it:
seen: set[str] = set()
for r in incoming:
if r.job_id in seen:
log.warning("duplicate result for %s; ignoring second copy", r.job_id)
continue
seen.add(r.job_id)
results[idx_map[r.job_id]] = r
The log output for the replayed webhook:
WARNING:merge:duplicate result for J-102; ignoring second copy
Three strategies exist for the write itself, and the choice matters:
| Strategy | Behavior | Use it when |
|---|---|---|
| Last-write-wins | Later copy overwrites the slot | Payloads are snapshots and newer is strictly better (fresh prices, live inventory) |
| First-write-wins | Later copies are dropped | First delivery is canonical; retries are replay noise |
| Explicit flagging | Keep one copy, record the extra count | You bill per result, dedupe downstream, or audit delivery |
My default is first-write-wins plus an explicit flag, and I’ll defend it: duplicate deliveries are usually byte-identical replays, so overwriting gains nothing — and it invites a race where a retry carrying a truncated or partially rendered payload clobbers a good one. If you cannot prove your replays are identical, flag them and diff the payloads offline. What you should never do is let the second copy silently win with no record that it happened. A duplicate count that suddenly spikes from zero to dozens is a signal your webhook receiver is timing out, and that diagnosis is worth more than either payload.
Putting It Together: A Complete KeyedMerge Class for Batch Reports
Assembled into one component: register records the job-ID-to-URL pair (the live submit wrapper calls it after the API responds), merge_result fills slots defensively, and finalize emits an ordered, validated report with a status per job. The main() demo stubs the submission step — it registers the same five job IDs locally instead of calling the API — so the script runs end-to-end exactly as published, with no network or API key; the comment inside main() shows the block to swap in for a live run.
from __future__ import annotations
import asyncio
from dataclasses import dataclass
import httpx
API = "https://api.finedata.ai"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}
@dataclass
class JobResult:
job_id: str
status: str
content: str = ""
class KeyedMerge:
"""Turns unordered async results into an ordered, validated batch."""
def __init__(self) -> None:
self._url: dict[str, str] = {} # job_id -> url
self._order: list[str] = [] # submission order
self._slots: dict[str, JobResult] = {} # job_id -> first result kept
self._dupes: dict[str, int] = {} # job_id -> extra copies seen
def register(self, job_id: str, url: str) -> None:
"""Record a dispatched job. Must run before any result arrives."""
if job_id in self._url:
raise ValueError(f"job_id {job_id} dispatched twice")
self._url[job_id] = url
self._order.append(job_id)
async def submit(self, client: httpx.AsyncClient, url: str) -> str:
resp = await client.post(
f"{API}/api/v1/async/scrape",
headers=HEADERS,
json={"url": url, "formats": ["markdown"]},
)
resp.raise_for_status()
job_id = resp.json()["job_id"]
self.register(job_id, url)
return job_id
def merge_result(self, result: JobResult) -> str:
job_id = result.job_id
if job_id not in self._url:
raise KeyError(
f"orphan result {job_id}: never dispatched by this pipeline"
)
if job_id in self._slots:
self._dupes[job_id] = self._dupes.get(job_id, 0) + 1
return "duplicate"
self._slots[job_id] = result
return "ok"
def finalize(self) -> list[dict]:
report = []
for pos, job_id in enumerate(self._order):
r = self._slots.get(job_id)
extra = self._dupes.get(job_id, 0)
if r is None:
status = "missing"
elif extra:
status = "duplicate"
else:
status = "ok"
report.append({
"position": pos,
"job_id": job_id,
"url": self._url[job_id],
"status": status,
"extra_copies": extra,
"content": r.content if r else None,
})
return report
async def main() -> None:
merge = KeyedMerge()
urls = [f"https://store.example.com/products/{n}" for n in range(101, 106)]
# Stubbed submission: register deterministic job ids locally so this
# script runs end-to-end as published (no network, no API key).
# For a live run, replace this loop with:
# async with httpx.AsyncClient(timeout=30) as client:
# await asyncio.gather(*(merge.submit(client, u) for u in urls))
# and feed merge.merge_result() from real webhook deliveries carrying
# the ids the API actually returned.
for i, url in enumerate(urls, start=101):
merge.register(f"J-{i}", url)
# Arrival simulation: scrambled order, one replayed webhook (J-102
# delivered twice), one job that never comes back (J-103).
incoming = [
JobResult("J-104", "completed", "markdown for 104"),
JobResult("J-101", "completed", "markdown for 101"),
JobResult("J-105", "completed", "markdown for 105"),
JobResult("J-102", "completed", "markdown for 102"),
JobResult("J-102", "completed", "markdown for 102"), # webhook retry
]
for r in incoming:
merge.merge_result(r)
for row in merge.finalize():
print(f"{row['position']} {row['job_id']} {row['url']} {row['status']}")
if __name__ == "__main__":
asyncio.run(main())
The end-to-end output, in submission order regardless of how the results arrived:
0 J-101 https://store.example.com/products/101 ok
1 J-102 https://store.example.com/products/102 duplicate
2 J-103 https://store.example.com/products/103 missing
3 J-104 https://store.example.com/products/104 ok
4 J-105 https://store.example.com/products/105 ok
Slot 2 is None in the array, not a shifted row from slot 3. The duplicate on slot 1 is visible rather than absorbed. Downstream, the report writer can decide what a missing row means — re-drive it, skip it, or fail the batch — without guessing at alignment.
One boundary this class assumes: single-threaded arrival processing. Inside one asyncio event loop, that assumption holds for free — merge_result contains no await, so no other task can interleave between the duplicate check and the slot write, making the whole method effectively atomic under cooperative scheduling. If your receiver handles deliveries on threads (a sync WSGI app, or run_in_executor offloads), wrap merge_result in a threading.Lock, because the check-then-write pair is not atomic across threads even though each individual dict operation is. And if you run multiple receiver processes, the map and the dedupe state must move to shared storage, where an idempotent write (a SET NX per job ID, for example) plays the role of the duplicate check. The keyed-merge pattern survives all three deployments; the bare in-process dict does not.
Measuring the Merge Overhead: Cost of Keyed Merge vs. Naive Append
The obvious objection: a dict lookup and a set update per result must cost something. Measure it instead of arguing about it. The baseline below models what a plain webhook receiver actually does — one append per delivery, in arrival order — rather than a bulk list copy, which would flatter the naive path:
import time
from types import SimpleNamespace
def bench(n: int, runs: int = 7) -> tuple[float, float]:
ids = [f"J-{i}" for i in range(n)]
idx_map = {j: i for i, j in enumerate(ids)}
incoming = [SimpleNamespace(job_id=j) for j in reversed(ids)]
best_append = best_merge = float("inf")
for _ in range(runs):
t0 = time.perf_counter()
naive = [] # what a plain receiver does:
for r in incoming: # one append per delivery,
naive.append(r) # in arrival order
t1 = time.perf_counter()
results = [None] * n # keyed merge
seen = set()
for r in incoming:
if r.job_id in seen:
continue
seen.add(r.job_id)
results[idx_map[r.job_id]] = r
t2 = time.perf_counter()
best_append = min(best_append, (t1 - t0) * 1000)
best_merge = min(best_merge, (t2 - t1) * 1000)
return best_append, best_merge
for n in (100, 1_000, 10_000):
a, m = bench(n)
print(f"{n:>6} append={a:8.3f} ms merge={m:8.3f} ms")
On my laptop (CPython 3.12, best of seven runs):
| Batch size | Per-delivery append (ms) | Keyed merge (ms) | Per-job overhead |
|---|---|---|---|
| 100 | 0.008 | 0.019 | ~0.11 µs |
| 1,000 | 0.072 | 0.184 | ~0.11 µs |
| 10,000 | 0.71 | 1.87 | ~0.12 µs |
The numbers are boring, which is the point. The keyed merge costs roughly a tenth of a microsecond more per result than the plain append, and that per-job overhead stays flat from 100 to 10,000 results — the merge does constant work per arrival (one dict probe, one set update, one list write), so total cost is linear with no hidden superlinearity. At webhook rates this is unmeasurable: a receiver processing fifty deliveries per second accrues about five microseconds of merge overhead per second, which is noise next to the JSON parse of a single payload, let alone the network round-trip that carried it. Even the 10,000-result batch merges in under two milliseconds — less time than one webhook delivery takes on the wire. Unless you are consolidating results from many receiver processes into one, per-job overhead is not a defensible reason to skip keyed merging; the first silently misaligned row costs more than every benchmark in this table combined.
Related Articles
Batch a URL List Into Partitioned Parquet Files
Turn a list of URLs into partitioned Parquet files with a staged Python pipeline: fetch, extract, validate, and load, with a run manifest you can audit.
TutorialRe-Run a Scrape Batch Without Duplicate Rows
Run a batch of URLs through a scrape pipeline and load results with idempotent upserts, so a re-run updates rows instead of duplicating them.
TutorialBuild a Three-Layer Scraping Pipeline: Raw, Clean, Curated
Build a scraping pipeline in three layers: raw HTML landing zone, cleaned records, curated dataset. Re-run extraction without re-fetching a page.