Your Batch 429s Are Self-Inflicted: Fix the Polling Storm
Your batch jobs fail with 429s at scale, but the scrapes are fine. Learn to diagnose self-inflicted rate limits from status polling and fix them.
On this page
The Puzzle: A Failing Batch and a Healthy Dashboard
Your overnight batch run died with 37 of 100 jobs marked failed, every single one an HTTP 429. The success-rate dashboard for the target domain looks fine. Even weirder: while the batch was melting down, a hand-rolled curl from the same machine returned 200. You restarted the run. It failed again, at a different job count.
Here is the uncomfortable part: the scrapes were never the problem. The status polling was. This post walks through the diagnosis, the instrumentation that proves it, and three fixes that turn a 37% failure rate into zero without touching a single page fetch.
The Symptom: Batch Scrapes Fail While Individual Requests Succeed
Start with the contradiction. One request, submitted by hand, works:
$ curl -s -o /dev/null -w "%{http_code}\n" \
-X POST https://store.example.com/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://store.example.com/products/1"}'
200
The batch run, minutes later, produces this:
12:04:01 worker-03 POST /scrape products/51 -> 200 job=a91f
12:04:01 worker-07 GET /jobs/a91f/status -> 200 running
12:04:02 worker-19 POST /scrape products/52 -> 429 rate limit: 50 req/s
12:04:02 worker-44 GET /jobs/b32c/status -> 429 rate limit: 50 req/s
12:04:03 worker-02 GET /jobs/b32c/status -> 429 rate limit: 50 req/s
...
12:04:41 worker-50 GET /jobs/e11d/status -> 429 rate limit: 50 req/s
12:04:41 batch aborted: 37/100 jobs marked failed
Read the log closely. The 429s are not landing on the page fetches — they land on POST /scrape submits and, mostly, on GET /jobs/{id}/status. The pages that did get submitted scraped fine. The failures cluster exactly where your client is loudest.
Before you blame the target or rotate proxies, separate the two hypotheses:
| Target is rate limiting you | You are rate limiting yourself |
|---|---|
| 429s scale with pages fetched from one domain | 429s scale with jobs in flight, regardless of domain |
| A lone test request starts failing mid-batch too | The lone request returns 200 while the batch burns |
| 429 body comes from the origin (challenge page, Retry-After set by the site) | 429s come from the gateway in front of the job queue, with a limit shared across all endpoints |
| Failures cluster on one problem domain | Failures spread evenly across every domain in the batch |
| Backoff on the scrape path helps | Backoff on the scrape path changes nothing — you were never throttled on scrapes |
If your symptoms live in the right column, stop tuning proxies. Your client is DDoSing its own request budget.
Why Status Polling Multiplies Your Request Count 10x
The async scrape pattern is: submit a job, poll until done, collect the result. The page fetch executes server-side and doesn’t touch your rate limit. The polling does. Here is the loop that most of us have written at least once:
import time
import requests
def run_job(url):
resp = requests.post("https://store.example.com/scrape",
json={"url": url}, timeout=10)
job_id = resp.json()["job_id"]
while True:
status = requests.get(
f"https://store.example.com/jobs/{job_id}/status", timeout=5
).json()
if status["state"] == "done":
return status["result"]
if status["state"] == "failed":
raise RuntimeError(f"job {job_id} failed")
time.sleep(0.5) # the offender
One bug is already visible above and the log proves it: when the submit itself is throttled — products/52 at 12:04:02 — resp.json() returns a rate-limit body with no job_id, so this loop crashes or records a “failed” job that never ran. Hold that thought; fix 3 closes it.
Half a second feels polite. It is not, because politeness is a property of total request rate, not of any one loop. Run the arithmetic for a realistic batch:
Jobs in flight: 200
Poll frequency: 2 req/s per job
Poll traffic: 400 req/s
Your API rate budget: 50 req/s
Overage: 8x, before counting the 200 submits
A single job that takes 15 seconds to render costs 1 submit plus 30 polls. One logical operation, 31 HTTP requests. And it compounds: every 429’d poll that your client retries adds another request, and plenty of clients treat a 429 on a status check as a failed job, which triggers a resubmit — a fresh page fetch for no reason.
One more detail that trips people up. If you drive an async scraping API like FineData, the status call hits the same front door and counts against the same request budget as the submit. The queue does the expensive work; the polls are often the majority of everything your key emits. Your dashboard says “100 pages scraped.” Your rate limiter saw 1,700 requests.
Measuring the Storm: Counting Real Requests Per Batch Item
Opinions about request volume are worthless. Instrument the client so every HTTP call is attributed to the logical operation that caused it, then read the numbers. You need a wrapper, a per-job counter, and five minutes.
import collections
import re
import time
import requests
PER_JOB = {} # url -> Counter
# Match the job-status endpoint exactly, not any URL that happens
# to contain "/jobs" (e.g. a scrape target like /careers/jobs/123).
STATUS_PATH = re.compile(r"^/jobs/[^/]+/status$")
def classify(url):
from urllib.parse import urlparse
return "status_poll" if STATUS_PATH.match(urlparse(url).path) else "scrape"
def tracked(method, url, url_key, **kwargs):
kwargs.setdefault("timeout", 10) # never let a hung endpoint stall the loop
resp = requests.request(method, url, **kwargs)
c = PER_JOB.setdefault(url_key, collections.Counter())
c[classify(url)] += 1 # count the request we just sent
if resp.status_code == 429:
c["429_hits"] += 1 # annotation only: marks requests that were
# throttled; NOT an additional request
return resp
def run_job_measured(url):
r = tracked("POST", "https://store.example.com/scrape", url, json={"url": url})
r.raise_for_status() # a 429'd submit surfaces here — same bug as products/52
job_id = r.json()["job_id"]
while True:
s = tracked("GET", f"https://store.example.com/jobs/{job_id}/status", url)
if s.status_code == 429:
time.sleep(0.5) # mirrors the original client: measure it, don't fix it
continue
state = s.json()["state"]
if state == "done":
break
if state == "failed":
raise RuntimeError(f"job {job_id} ({url}) failed") # surfaced, not swallowed
time.sleep(0.5)
c = PER_JOB[url]
extras = f", 429: {c['429_hits']}" if c["429_hits"] else ""
print(f"{url}: scrape: {c['scrape']} req, status_poll: {c['status_poll']} req{extras}")
def batch_report():
totals = collections.Counter()
for c in PER_JOB.values():
totals.update(c)
jobs = len(PER_JOB)
# '429_hits' marks how many of the requests above came back throttled;
# it is not an extra request, so it stays out of the average.
sent = totals["scrape"] + totals["status_poll"]
print("---- batch totals ----")
print(f"jobs: {jobs}")
print(f"scrape submits: {totals['scrape']}")
print(f"status polls: {totals['status_poll']}")
print(f"requests sent: {sent}")
print(f"avg per job: {sent / jobs:.1f}")
print(f"429s received: {totals['429_hits']}")
Run this against a 100-job batch and the numbers stop being theoretical. A typical report from the broken client looks like:
---- batch totals ----
jobs: 100
scrape submits: 100
status polls: 1,540
requests sent: 1,640
avg per job: 16.4
429s received: 37
Sixteen requests per page scraped, and the polls outnumber the submits 15-to-1. That is the storm. Now the fixes.
Fix 1: Exponential Poll Backoff
The 0.5-second fixed poll interval is the root cause. A job that takes 20 seconds to render does not need 40 status checks — it needs maybe four. Poll on an exponential schedule instead, and cap the interval so long jobs still finish promptly:
import time
def poll_with_backoff(job_id, url_key, first=1.0, factor=1.7, cap=15.0):
"""Poll job status with exponential backoff: 1s, 1.7s, 2.9s, ... capped at 15s."""
delay = first
while True:
s = tracked("GET", f"https://store.example.com/jobs/{job_id}/status", url_key)
if s.status_code == 429:
# The gateway is telling us we're loud. Back off harder than
# the normal schedule instead of hammering at the same cadence.
delay = min(delay * factor, cap)
time.sleep(delay)
continue
state = s.json()["state"]
if state == "done":
return s.json()["result"]
if state == "failed":
raise RuntimeError(f"job {job_id} ({url_key}) failed")
time.sleep(delay)
delay = min(delay * factor, cap)
The same 20-second job now costs 1 submit plus 7 polls instead of 41. Across 200 concurrent jobs, poll traffic drops from 400 req/s to roughly 30 req/s — under budget with headroom for the submits. If the API returns a Retry-After header on its 429s, honor it: time.sleep(float(s.headers.get("Retry-After", delay))).
Fix 2: A Central Request-Budget Token Bucket
Backoff fixes the polling cadence, but each worker still decides its own sleep in isolation. With 200 workers, “polite” per-worker intervals can still sum to an impolite total. Move the budget to one place: a token bucket that every request — submit or poll — must pass through.
import threading
import time
class TokenBucket:
"""Thread-safe rate limiter shared by all workers.
refill_rate is in tokens (requests) per second; capacity is the
burst size. Every HTTP call must acquire one token before sending.
"""
def __init__(self, capacity, refill_rate):
self.capacity = capacity
self.refill_rate = refill_rate
self.tokens = capacity
self.lock = threading.Lock()
self.last = time.monotonic()
def acquire(self):
with self.lock:
now = time.monotonic()
self.tokens = min(self.capacity,
self.tokens + (now - self.last) * self.refill_rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return
wait = (1 - self.tokens) / self.refill_rate
time.sleep(wait) # sleep outside the lock
self.acquire()
# Budget: 50 req/s shared across submits AND polls, with a small burst
# allowance so a fresh batch can start without queueing on token one.
BUCKET = TokenBucket(capacity=10, refill_rate=45.0)
def budgeted(method, url, url_key, **kwargs):
BUCKET.acquire()
return tracked(method, url, url_key, **kwargs)
Set refill_rate slightly below the gateway’s published limit (45 of 50 req/s here) so transient clock skew on their side doesn’t cost you. The burst capacity absorbs the initial submit wave; after that, the bucket — not the worker count — decides your request rate. Add or remove workers freely: total traffic stays at 45 req/s no matter what.
Fix 3: Handle 429s on the Submit Path
Fixes 1 and 2 make 429s rare, but rare is not never, and the original client had a second, quieter bug: a throttled submit returned a rate-limit body, resp.json()["job_id"] raised a KeyError, and the batch recorded a “failed” job that never ran. Treat a 429 on the submit as “try again shortly,” not “job failed”:
import time
def submit_with_retry(url, url_key, max_attempts=5):
"""Submit a scrape job, retrying with backoff if the submit is throttled."""
delay = 1.0
for attempt in range(max_attempts):
r = budgeted("POST", "https://store.example.com/scrape", url, json={"url": url})
if r.status_code == 429:
retry_after = float(r.headers.get("Retry-After", delay))
time.sleep(retry_after)
delay = min(delay * 2, 30.0)
continue
r.raise_for_status() # non-429 errors (4xx/5xx) are real failures
return r.json()["job_id"]
raise RuntimeError(f"submit for {url} still throttled after {max_attempts} attempts")
Two things matter here. First, raise_for_status() after the 429 check means real errors — a 400 for a malformed URL, a 500 on their side — still surface instead of being retried into oblivion. Second, the retry honors Retry-After when the gateway sends it, and falls back to doubling delays when it doesn’t. A throttled submit now costs a few seconds of waiting, not a phantom failed job.
The Result: Zero Failures, Same Data, Fewer Requests
Wire the three fixes together — submit_with_retry for the submit, poll_with_backoff for the status loop, everything through budgeted — and rerun the instrumented batch. The report from the fixed client:
---- batch totals ----
jobs: 100
scrape submits: 100
status polls: 412
requests sent: 512
avg per job: 5.1
429s received: 0
Same 100 pages scraped. Total requests down 69%, 429s down to zero, and the batch completes faster than before because workers no longer burn time retrying throttled polls. The dashboard was right all along — the target never had a problem. Your client did, and the polling loop was where it lived.
The takeaway generalizes past scraping. Anywhere you have an async API with a shared rate limit, the request you make while waiting is the one that gets you throttled. Measure the real request count per logical operation before you tune anything, budget globally rather than per worker, and never let a throttled submit masquerade as a failed job.
Related Articles
Your Rate Limiter Multiplies Across Workers: Fix It
You set one request per second but still get 429s. Each worker runs its own limiter, so your real rate is N times the budget. Here's how to fix it.
TutorialYour Scraper Died With Exit Code 137: Fix the OOM Kill
Workers die mid-batch with exit code 137 and no logged exception. Learn to confirm the OOM kill, find the memory hog, and stop buffering full page bodies.
TutorialYour Backoff Ignores Retry-After: Fix Persistent 429s
Still throttled despite exponential backoff? Learn to parse Retry-After headers, cap per-host concurrency, and stop the 429 cycle stalling your scrape pipeline.