Skip to content
Tutorial 10 min read

Your Batch 429s Are Self-Inflicted: Fix the Polling Storm

Your batch jobs fail with 429s at scale, but the scrapes are fine. Learn to diagnose self-inflicted rate limits from status polling and fix them.

FE
FineData Engineering · Editorial Policy
|
On this page

The Puzzle: A Failing Batch and a Healthy Dashboard

Your overnight batch run died with 37 of 100 jobs marked failed, every single one an HTTP 429. The success-rate dashboard for the target domain looks fine. Even weirder: while the batch was melting down, a hand-rolled curl from the same machine returned 200. You restarted the run. It failed again, at a different job count.

Here is the uncomfortable part: the scrapes were never the problem. The status polling was. This post walks through the diagnosis, the instrumentation that proves it, and three fixes that turn a 37% failure rate into zero without touching a single page fetch.

The Symptom: Batch Scrapes Fail While Individual Requests Succeed

Start with the contradiction. One request, submitted by hand, works:

$ curl -s -o /dev/null -w "%{http_code}\n" \
    -X POST https://store.example.com/scrape \
    -H "Content-Type: application/json" \
    -d '{"url": "https://store.example.com/products/1"}'
200

The batch run, minutes later, produces this:

12:04:01 worker-03  POST /scrape           products/51  -> 200  job=a91f
12:04:01 worker-07  GET  /jobs/a91f/status              -> 200  running
12:04:02 worker-19  POST /scrape           products/52  -> 429  rate limit: 50 req/s
12:04:02 worker-44  GET  /jobs/b32c/status              -> 429  rate limit: 50 req/s
12:04:03 worker-02  GET  /jobs/b32c/status              -> 429  rate limit: 50 req/s
...
12:04:41 worker-50  GET  /jobs/e11d/status              -> 429  rate limit: 50 req/s
12:04:41 batch aborted: 37/100 jobs marked failed

Read the log closely. The 429s are not landing on the page fetches — they land on POST /scrape submits and, mostly, on GET /jobs/{id}/status. The pages that did get submitted scraped fine. The failures cluster exactly where your client is loudest.

Before you blame the target or rotate proxies, separate the two hypotheses:

Target is rate limiting youYou are rate limiting yourself
429s scale with pages fetched from one domain429s scale with jobs in flight, regardless of domain
A lone test request starts failing mid-batch tooThe lone request returns 200 while the batch burns
429 body comes from the origin (challenge page, Retry-After set by the site)429s come from the gateway in front of the job queue, with a limit shared across all endpoints
Failures cluster on one problem domainFailures spread evenly across every domain in the batch
Backoff on the scrape path helpsBackoff on the scrape path changes nothing — you were never throttled on scrapes

If your symptoms live in the right column, stop tuning proxies. Your client is DDoSing its own request budget.

Why Status Polling Multiplies Your Request Count 10x

The async scrape pattern is: submit a job, poll until done, collect the result. The page fetch executes server-side and doesn’t touch your rate limit. The polling does. Here is the loop that most of us have written at least once:

import time
import requests

def run_job(url):
    resp = requests.post("https://store.example.com/scrape",
                         json={"url": url}, timeout=10)
    job_id = resp.json()["job_id"]

    while True:
        status = requests.get(
            f"https://store.example.com/jobs/{job_id}/status", timeout=5
        ).json()
        if status["state"] == "done":
            return status["result"]
        if status["state"] == "failed":
            raise RuntimeError(f"job {job_id} failed")
        time.sleep(0.5)  # the offender

One bug is already visible above and the log proves it: when the submit itself is throttled — products/52 at 12:04:02 — resp.json() returns a rate-limit body with no job_id, so this loop crashes or records a “failed” job that never ran. Hold that thought; fix 3 closes it.

Half a second feels polite. It is not, because politeness is a property of total request rate, not of any one loop. Run the arithmetic for a realistic batch:

Jobs in flight:            200
Poll frequency:            2 req/s per job
Poll traffic:              400 req/s
Your API rate budget:      50 req/s
Overage:                   8x, before counting the 200 submits

A single job that takes 15 seconds to render costs 1 submit plus 30 polls. One logical operation, 31 HTTP requests. And it compounds: every 429’d poll that your client retries adds another request, and plenty of clients treat a 429 on a status check as a failed job, which triggers a resubmit — a fresh page fetch for no reason.

One more detail that trips people up. If you drive an async scraping API like FineData, the status call hits the same front door and counts against the same request budget as the submit. The queue does the expensive work; the polls are often the majority of everything your key emits. Your dashboard says “100 pages scraped.” Your rate limiter saw 1,700 requests.

Measuring the Storm: Counting Real Requests Per Batch Item

Opinions about request volume are worthless. Instrument the client so every HTTP call is attributed to the logical operation that caused it, then read the numbers. You need a wrapper, a per-job counter, and five minutes.

import collections
import re
import time

import requests

PER_JOB = {}  # url -> Counter

# Match the job-status endpoint exactly, not any URL that happens
# to contain "/jobs" (e.g. a scrape target like /careers/jobs/123).
STATUS_PATH = re.compile(r"^/jobs/[^/]+/status$")

def classify(url):
    from urllib.parse import urlparse
    return "status_poll" if STATUS_PATH.match(urlparse(url).path) else "scrape"

def tracked(method, url, url_key, **kwargs):
    kwargs.setdefault("timeout", 10)  # never let a hung endpoint stall the loop
    resp = requests.request(method, url, **kwargs)
    c = PER_JOB.setdefault(url_key, collections.Counter())
    c[classify(url)] += 1  # count the request we just sent
    if resp.status_code == 429:
        c["429_hits"] += 1  # annotation only: marks requests that were
                           # throttled; NOT an additional request
    return resp

def run_job_measured(url):
    r = tracked("POST", "https://store.example.com/scrape", url, json={"url": url})
    r.raise_for_status()  # a 429'd submit surfaces here — same bug as products/52
    job_id = r.json()["job_id"]

    while True:
        s = tracked("GET", f"https://store.example.com/jobs/{job_id}/status", url)
        if s.status_code == 429:
            time.sleep(0.5)  # mirrors the original client: measure it, don't fix it
            continue
        state = s.json()["state"]
        if state == "done":
            break
        if state == "failed":
            raise RuntimeError(f"job {job_id} ({url}) failed")  # surfaced, not swallowed
        time.sleep(0.5)

    c = PER_JOB[url]
    extras = f", 429: {c['429_hits']}" if c["429_hits"] else ""
    print(f"{url}: scrape: {c['scrape']} req, status_poll: {c['status_poll']} req{extras}")

def batch_report():
    totals = collections.Counter()
    for c in PER_JOB.values():
        totals.update(c)
    jobs = len(PER_JOB)
    # '429_hits' marks how many of the requests above came back throttled;
    # it is not an extra request, so it stays out of the average.
    sent = totals["scrape"] + totals["status_poll"]
    print("---- batch totals ----")
    print(f"jobs:              {jobs}")
    print(f"scrape submits:    {totals['scrape']}")
    print(f"status polls:      {totals['status_poll']}")
    print(f"requests sent:     {sent}")
    print(f"avg per job:       {sent / jobs:.1f}")
    print(f"429s received:     {totals['429_hits']}")

Run this against a 100-job batch and the numbers stop being theoretical. A typical report from the broken client looks like:

---- batch totals ----
jobs:              100
scrape submits:    100
status polls:      1,540
requests sent:    1,640
avg per job:       16.4
429s received:     37

Sixteen requests per page scraped, and the polls outnumber the submits 15-to-1. That is the storm. Now the fixes.

Fix 1: Exponential Poll Backoff

The 0.5-second fixed poll interval is the root cause. A job that takes 20 seconds to render does not need 40 status checks — it needs maybe four. Poll on an exponential schedule instead, and cap the interval so long jobs still finish promptly:

import time

def poll_with_backoff(job_id, url_key, first=1.0, factor=1.7, cap=15.0):
    """Poll job status with exponential backoff: 1s, 1.7s, 2.9s, ... capped at 15s."""
    delay = first
    while True:
        s = tracked("GET", f"https://store.example.com/jobs/{job_id}/status", url_key)
        if s.status_code == 429:
            # The gateway is telling us we're loud. Back off harder than
            # the normal schedule instead of hammering at the same cadence.
            delay = min(delay * factor, cap)
            time.sleep(delay)
            continue
        state = s.json()["state"]
        if state == "done":
            return s.json()["result"]
        if state == "failed":
            raise RuntimeError(f"job {job_id} ({url_key}) failed")
        time.sleep(delay)
        delay = min(delay * factor, cap)

The same 20-second job now costs 1 submit plus 7 polls instead of 41. Across 200 concurrent jobs, poll traffic drops from 400 req/s to roughly 30 req/s — under budget with headroom for the submits. If the API returns a Retry-After header on its 429s, honor it: time.sleep(float(s.headers.get("Retry-After", delay))).

Fix 2: A Central Request-Budget Token Bucket

Backoff fixes the polling cadence, but each worker still decides its own sleep in isolation. With 200 workers, “polite” per-worker intervals can still sum to an impolite total. Move the budget to one place: a token bucket that every request — submit or poll — must pass through.

import threading
import time

class TokenBucket:
    """Thread-safe rate limiter shared by all workers.

    refill_rate is in tokens (requests) per second; capacity is the
    burst size. Every HTTP call must acquire one token before sending.
    """
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity
        self.refill_rate = refill_rate
        self.tokens = capacity
        self.lock = threading.Lock()
        self.last = time.monotonic()

    def acquire(self):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity,
                              self.tokens + (now - self.last) * self.refill_rate)
            self.last = now
            if self.tokens >= 1:
                self.tokens -= 1
                return
            wait = (1 - self.tokens) / self.refill_rate
        time.sleep(wait)  # sleep outside the lock
        self.acquire()

# Budget: 50 req/s shared across submits AND polls, with a small burst
# allowance so a fresh batch can start without queueing on token one.
BUCKET = TokenBucket(capacity=10, refill_rate=45.0)

def budgeted(method, url, url_key, **kwargs):
    BUCKET.acquire()
    return tracked(method, url, url_key, **kwargs)

Set refill_rate slightly below the gateway’s published limit (45 of 50 req/s here) so transient clock skew on their side doesn’t cost you. The burst capacity absorbs the initial submit wave; after that, the bucket — not the worker count — decides your request rate. Add or remove workers freely: total traffic stays at 45 req/s no matter what.

Fix 3: Handle 429s on the Submit Path

Fixes 1 and 2 make 429s rare, but rare is not never, and the original client had a second, quieter bug: a throttled submit returned a rate-limit body, resp.json()["job_id"] raised a KeyError, and the batch recorded a “failed” job that never ran. Treat a 429 on the submit as “try again shortly,” not “job failed”:

import time

def submit_with_retry(url, url_key, max_attempts=5):
    """Submit a scrape job, retrying with backoff if the submit is throttled."""
    delay = 1.0
    for attempt in range(max_attempts):
        r = budgeted("POST", "https://store.example.com/scrape", url, json={"url": url})
        if r.status_code == 429:
            retry_after = float(r.headers.get("Retry-After", delay))
            time.sleep(retry_after)
            delay = min(delay * 2, 30.0)
            continue
        r.raise_for_status()  # non-429 errors (4xx/5xx) are real failures
        return r.json()["job_id"]
    raise RuntimeError(f"submit for {url} still throttled after {max_attempts} attempts")

Two things matter here. First, raise_for_status() after the 429 check means real errors — a 400 for a malformed URL, a 500 on their side — still surface instead of being retried into oblivion. Second, the retry honors Retry-After when the gateway sends it, and falls back to doubling delays when it doesn’t. A throttled submit now costs a few seconds of waiting, not a phantom failed job.

The Result: Zero Failures, Same Data, Fewer Requests

Wire the three fixes together — submit_with_retry for the submit, poll_with_backoff for the status loop, everything through budgeted — and rerun the instrumented batch. The report from the fixed client:

---- batch totals ----
jobs:              100
scrape submits:    100
status polls:      412
requests sent:     512
avg per job:       5.1
429s received:     0

Same 100 pages scraped. Total requests down 69%, 429s down to zero, and the batch completes faster than before because workers no longer burn time retrying throttled polls. The dashboard was right all along — the target never had a problem. Your client did, and the polling loop was where it lived.

The takeaway generalizes past scraping. Anywhere you have an async API with a shared rate limit, the request you make while waiting is the one that gets you throttled. Measure the real request count per logical operation before you tune anything, budget globally rather than per worker, and never let a throttled submit masquerade as a failed job.

#polling #async-jobs #rate-limiting #exponential-backoff #observability #slot:operations

Related Articles