Your Backoff Ignores Retry-After: Fix Persistent 429s

The article with the fix applied

The Symptom: A Scrape Log Full of 429s

The tail of your scrape log reads like a broken record:

worker-03 GET https://store.example.com/products/sku-4412 -> 429
  retry 1 in 2.0s
worker-03 GET https://store.example.com/products/sku-4412 -> 429  (Retry-After: 30)
  retry 2 in 4.0s
worker-03 GET https://store.example.com/products/sku-4412 -> 429  (Retry-After: 30)
  retry 3 in 8.0s

The server told the worker to wait 30 seconds. The worker waited two. That single ignored header, multiplied across 16 workers and a few thousand URLs, is why your pipeline spends its evening collecting 429s instead of pages. Backoff is implemented, jitter is in place, and the cycle still repeats — because the code guesses at a cooldown the server already published.

Why Exponential Backoff Keeps Colliding with the Server’s Throttle Window

Exponential backoff comes from a world where nobody knows the global state: contention on a shared bus, leadership election, lock retries. Randomized doubling exists there to spread out blind attempts. A per-client rate limiter is a different animal. It is deterministic, it knows exactly when your penalty expires, and — if you’re lucky — it prints that timestamp on every rejection.

Guessing anyway is worse than pointless. Every request you fire inside the penalty window can extend it, since many limiters count rejected requests against your budget. Even the friendly ones that don’t still cost you a connection, a TLS handshake, and a log line.

Here is one URL’s retry ladder against store.example.com, which throttles at 2 requests/second and returns Retry-After on every 429:

Attempt	Backoff sleep before it	Retry-After returned	Verdict
1	—	30	429; 30s too early
2	2s	30	429; 28s too early
3	4s	60 (window extended by the hits above)	429; 56s too early
4	8s	60	429; 52s too early
5	16s	60	429; 44s too early
6	32s	60	429; 28s too early
7	64s	—	200

Seven doomed requests and 126 seconds of sleeping to learn what the server said in a dozen bytes on response one. Obey the first header and the whole ladder collapses to a single 30-second wait.

Parsing Retry-After Correctly: Integer Seconds and HTTP-Date Formats

The header has exactly two legal forms: an integer number of seconds (Retry-After: 30) or an HTTP-date (Retry-After: Wed, 21 Oct ... GMT). Real servers also send floats, "30s", and empty strings, so parse defensively:

import email.utils
import random
from datetime import datetime, timezone

RETRY_AFTER_CAP_S = 300  # anything larger: skip and requeue, don't sleep

def parse_retry_after(raw, now=None):
    """Seconds to wait from a Retry-After header, or None if absent/garbage."""
    if raw is None:
        return None
    raw = str(raw).strip()

    # Form 1: delay-seconds. Tolerate "30.0" from sloppy origins.
    try:
        return max(0, int(float(raw)))
    except ValueError:
        pass

    # Form 2: HTTP-date, compared against our clock.
    try:
        when = email.utils.parsedate_to_datetime(raw)
        if when.tzinfo is None:
            when = when.replace(tzinfo=timezone.utc)
        now = now or datetime.now(timezone.utc)
        return max(0, int((when - now).total_seconds()))
    except (TypeError, ValueError):
        return None

def wait_seconds(attempt, retry_after):
    """Retry-After wins. Jittered exponential backoff is only the fallback."""
    if retry_after is not None and retry_after <= RETRY_AFTER_CAP_S:
        return retry_after * 1.25  # pad: limiters reset late more often than early
    guess = min(60.0, 1.0 * (2 ** attempt))
    return guess * (0.5 + random.random())  # full jitter

Two details worth arguing about. The 1.25 pad exists because a cooldown that says “30” frequently means “30 plus whatever slop the limiter adds when it resets”; I’d rather wait 38 seconds once than collect two more 429s. And the 300-second cap is non-negotiable: a misconfigured gateway returning Retry-After: 86400 should never park a worker for a day. Past the cap, skip the URL and requeue it.

The fallback path still matters. Plenty of CDN-fronted hosts emit 429s with no Retry-After at all — there, jittered exponential backoff is the correct tool, just demoted from default to last resort.

The Parallel-Worker Trap: Per-Worker Backoff Still Blows the Rate Budget

Fix one worker and the fleet still fails. Here’s a request log from a 16-worker pool, each worker sleeping 2 seconds between attempts against store.example.com:

worker-07 GET /products/sku-0231  -> 429 (retry in 2.0s)
worker-12 GET /products/sku-1104  -> 429 (retry in 2.0s)
worker-02 GET /products/sku-0059  -> 429 (retry in 2.0s)
worker-15 GET /products/sku-0877  -> 429 (retry in 2.0s)
worker-09 GET /products/sku-0346  -> 200
worker-04 GET /products/sku-0198  -> 429 (retry in 2.0s)

The arithmetic is brutal. One request every ~2 seconds per worker means each worker offers roughly 0.5 req/s. Sixteen workers offer about 8 req/s in aggregate.

Setting	Value
Workers	16
Per-worker interval	~2s (latency + backoff)
Aggregate offered rate	~8 req/s
store.example.com limit	2 req/s
Oversubscription	4x
Requests landing in the 429 bin	~75%

Per-worker backoff controls the wrong unit. The limit is per host, not per coroutine, and no amount of individual politeness fixes a fleet that is 4x over budget as a group. Worse, if every worker honors a 30-second Retry-After independently, all 16 wake at the same instant and stampede the newly reopened window together.

You need two pieces of shared state: an admission cap per hostname, and a cooldown every worker reads before dispatching.

Capping Per-Host Concurrency with a Hostname-Keyed Semaphore

A global semaphore on “the scrape pool” is the usual first attempt, and it’s the wrong shape — one slow host then throttles your requests to every other host. Key the semaphore by hostname instead:

import asyncio
from urllib.parse import urlsplit

class HostGates:
    """Per-hostname admission caps, shared across the whole pool."""
    def __init__(self, default_cap=4, caps=None):
        self.default_cap = default_cap
        self.caps = caps or {}
        self._sems: dict[str, asyncio.Semaphore] = {}

    def gate(self, url: str) -> asyncio.Semaphore:
        host = (urlsplit(url).hostname or "").lower()
        if host not in self._sems:
            cap = self.caps.get(host, self.default_cap)
            self._sems[host] = asyncio.Semaphore(cap)
        return self._sems[host]

MAX_INFLIGHT = {
    "example.com": 6,        # blog host, generous limit
    "store.example.com": 2, # the storefront that throttles
    "api.example.com": 8,
}

The worker pool then acquires its host’s gate around the fetch:

import asyncio, httpx

async def worker(queue, client, gates, board):
    while True:
        url = await queue.get()
        try:
            host = urlsplit(url).hostname.lower()
            wait = board.remaining(host)          # section below
            if wait > 0:
                await asyncio.sleep(wait + random.random())  # jittered wake
            async with gates.gate(url):
                resp = await client.get(url)
                await handle(resp, host, board)
        finally:
            queue.task_done()

async def run(urls):
    gates = HostGates(default_cap=4, caps=MAX_INFLIGHT)
    board = CooldownBoard()
    queue = asyncio.Queue()
    for u in urls:
        queue.put_nowait(u)
    async with httpx.AsyncClient(timeout=30) as client:
        await asyncio.gather(*[worker(queue, client, gates, board)
                               for _ in range(16)])

One caveat I want to be explicit about, because it bites everyone: a semaphore bounds concurrency, not rate. If store.example.com responds in 300ms, a cap of 2 still yields ~6.7 req/s — three times the limit. If it responds in 1.5s, the same cap yields 1.3 req/s and you’re under. Caps must be tuned against measured latency.

My opinionated take: I prefer the two-knob setup (per-host cap plus the shared cooldown below) over a single global rate limiter. Under variable latency, a fixed req/s number is either wasteful or wrong, and the cap-plus-cooldown pair degrades gracefully. Reasonable people run a token bucket instead and sleep the difference; that works too, it’s just one more number to keep honest.

Sharing the Retry-After Cooldown Across All Workers

The semaphore bounds how many requests are in flight. The cooldown decides whether any should be. One worker’s 429 must pause the entire pool for that host — so the Retry-After value goes into shared state, not into a local variable:

import time

class CooldownBoard:
    """Per-host 'do not dispatch before' timestamps, visible to every worker."""
    def __init__(self):
        self._until: dict[str, float] = {}

    def extend(self, host: str, seconds: float) -> None:
        now = time.time()
        current = self._until.get(host, 0.0)
        # Largest observed wait wins; a fresh 429 never shortens a penalty.
        self._until[host] = max(current, now + seconds)

    def remaining(self, host: str) -> float:
        return max(0.0, self._until.get(host, 0.0) - time.time())

The 429 handler becomes three lines:

if resp.status_code == 429:
    ra = parse_retry_after(resp.headers.get("retry-after"))
    if ra is not None:
        board.extend(host, ra * 1.25)
    raise Throttled(url)  # worker sleeps on board.remaining(host) next loop

For a pool spread across processes or machines, the dict becomes a Redis key with a compare-and-set:

async def extend_shared(redis, host, seconds):
    key = f"cooldown:{host}"
    until = time.time() + seconds
    current = await redis.get(key)
    if current is None or float(current) < until:
        await redis.set(key, until)

The read-then-write race is benign — worst case two workers both extend, and the cooldown ends up slightly longer. Never wrap it in a lock; the whole point is to avoid coordination on the hot path.

Two choices here that throughput-minded readers will push back on. First, the board takes the maximum observed wait rather than the latest, so a straggler 429 with a big Retry-After can hold the pool even after smaller values arrive. I want that: the straggler usually reflects the real state of the limiter. Second, I do not clear the cooldown on a 200. A success proves the window reopened, but clearing invites all 16 workers to fire the instant one of them gets through. Let the timestamp expire on its own. You lose a few seconds of idle per incident; you avoid re-tripping the limiter you just climbed out of.

Bounding the Retry Storm: Retry Limits, Circuit Breaking, and Graceful Skips

Every mechanism above can still loop forever if the target keeps saying no. Bound it explicitly:

THROTTLE_POLICY = {
    "max_429_retries": 4,        # per URL, per run
    "retry_after_cap_s": 300,    # larger -> skip and requeue
    "cooldown_pad": 1.25,        # multiply server-provided waits
    "wake_jitter_s": 1.0,        # spread the herd on shared wakes
    "circuit": {
        "consecutive_429_threshold": 25,
        "open_seconds": 600,
    },
    "max_inflight": {
        "example.com": 6,
        "store.example.com": 2,
        "api.example.com": 8,
    },
}

Situation	Action
429, Retry-After ≤ 30s	Pad by cooldown_pad, sleep, retry in place
429, Retry-After 31–300s	Release the host slot, sleep off-slot, re-acquire, retry
429, Retry-After > 300s or absent	Skip URL; requeue with a not_before timestamp
429 on attempt > max_429_retries	Skip URL; requeue for the next run
≥ 25 consecutive 429s on one host	Open the circuit; halt every worker for that host
Circuit open past open_seconds	Half-open: one probe request; close on 200

The not_before field on requeued URLs matters more than it looks. A dead-letter