Your Rate Limiter Multiplies Across Workers: Fix It

How Four Workers Quietly Turned Your 1 req/s Budget into 4 req/s

The symptoms look like a haunting. You configured one request per second against store.example.com. The number is in the config, reviewed in the PR, deployed to production. Then someone scaled the service from one Gunicorn worker to four, and the 429s started within the hour. Your dashboards insist the client is sending four requests per second, even though nothing anywhere says four.

Here is the code that does it. It looks completely innocent:

# app.py
import httpx
from aiolimiter import AsyncLimiter

# Module scope. Evaluated once per process, at import time.
limiter = AsyncLimiter(1, 1)  # "one request per second"

async def fetch_order(order_id: int, client: httpx.AsyncClient):
    async with limiter:
        r = await client.get(f"https://store.example.com/api/orders/{order_id}")
        return r.json()

gunicorn -w 4 app:app uses a pre-fork model. The master imports your module, then forks four workers. Each fork inherits a copy of limiter — and after the fork, nothing about those copies is connected. Four processes, four independent token buckets, each happily permitting 1 req/s against the same host. uvicorn --workers 4 behaves identically. So do Celery workers, and so do Kubernetes replicas, which are just this bug with extra YAML.

The arithmetic is unkind:

Workers	Per-worker config	Actual combined rate	store.example.com budget (1 req/s)
1	1 req/s	1 req/s	within budget
2	1 req/s	2 req/s	2x over
4	1 req/s	4 req/s	4x over
8	1 req/s	8 req/s	8x over

Your test suite never caught it because pytest runs in one process. The bug only exists at the process boundary, which is exactly the boundary your tests don’t exercise. Every worker is individually compliant. The violation is a property of the group.

Reproduce the Bug Locally: Spawn 4 Workers and Watch the 429s Roll In

Before touching production, prove the multiplication to yourself. This script is self-contained: four processes, each sleeping a full second between requests, exactly like a correctly configured per-process limiter would produce. Point TARGET at an endpoint you control — a local counting stub works — or at the real thing in a staging window.

# repro.py — four processes, four limiters, one budget violated
import multiprocessing as mp
import os
import time
from datetime import datetime, timezone

import httpx

TARGET = "https://store.example.com/api/orders"
RUN_SECONDS = 10

def stamp() -> str:
    return datetime.now(timezone.utc).strftime("%H:%M:%S.%f")[:-3]

def worker(run_seconds: int):
    pid = os.getpid()
    deadline = time.monotonic() + run_seconds
    with httpx.Client(timeout=10) as client:
        while time.monotonic() < deadline:
            try:
                status = client.get(TARGET).status_code
            except httpx.HTTPError as exc:
                status = f"ERR:{type(exc).__name__}"
            print(f"{stamp()} pid={pid} -> {status}", flush=True)
            time.sleep(1.0)  # "my rate limit: one per second"

if __name__ == "__main__":
    procs = [mp.Process(target=worker, args=(RUN_SECONDS,)) for _ in range(4)]
    for p in procs:
        p.start()
    for p in procs:
        p.join()

Run it, and within two seconds you get output like this:

14:02:01.102 pid=4181 -> 200
14:02:01.233 pid=4182 -> 200
14:02:01.371 pid=4183 -> 429
14:02:01.498 pid=4184 -> 429
14:02:02.109 pid=4181 -> 429
14:02:02.244 pid=4182 -> 429
14:02:02.381 pid=4183 -> 429
14:02:02.515 pid=4184 -> 429

Four log lines land inside the same 400 ms window. Look at any single process in isolation and it behaves perfectly: one request, one-second sleep, repeat. No worker is doing anything wrong. The failure exists only in the combined view, which is why per-worker logs look clean while the server’s threat response disagrees.

Rule Out the Lookalikes: A 429 Differential Diagnosis Table

Not every surprise 429 is the multiplication bug, and treating the wrong disease wastes an afternoon. Four suspects cover most cases:

Suspect	Symptom pattern	Distinguishing evidence	Two-minute check
Per-worker limiters	429s begin right after worker or replica count changes	Combined rate equals configured rate × N	Rerun with one worker. If 429s vanish, it’s multiplication
Misconfigured limit value	429s since the first deploy, even with one worker	Single-worker test still gets refused	Print the limit value at runtime where the request is made, not from config
Retry-storm amplification	Requests far exceed jobs; errors breed more errors	Retry counters climb; outbound traffic spikes after the first failure	Count requests sent per successfully completed job
Window-boundary bursts	429s arrive in pairs at regular ticks	Two requests land milliseconds apart at window edges	Log inter-arrival deltas; bursts cluster at exact boundary offsets

Run the checks in that order. The single-worker run is cheapest and eliminates half the table immediately. Retry storms often ride on top of one of the other three, so fix the base rate first, then re-measure before assuming the retries are innocent. If your throughput decays gradually rather than failing hard, that’s usually a throttling signal worth a separate look before you blame the limiter at all.

Fix 1: Share One Token Bucket in Redis with an Atomic Lua Script

The fix is to move the counter out of process memory into something every worker can see. Redis is the default answer, but the detail that matters is atomicity: acquiring a token must be a check-and-decrement that no other worker can interleave with. If you implement it in Python as a GET followed by a conditional SET, two workers will read tokens=1 in the same millisecond, both will proceed, and your budget is breached anyway — a smaller bug, same family.

Redis executes a Lua script as one indivisible operation, so the whole read-refill-check-decrement happens with nobody else in between:

-- token_bucket.lua
-- KEYS[1] = bucket key, e.g. ratelimit:store.example.com
-- ARGV[1] = capacity (burst size)
-- ARGV[2] = refill rate, tokens per second
-- ARGV[3] = now, seconds since epoch (caller-supplied)
-- ARGV[4] = tokens requested
local key      = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill   = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local want     = tonumber(ARGV[4])

local state = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts     = tonumber(state[2])

if tokens == nil then
  tokens = capacity
  ts = now
end

local elapsed = math.max(0, now - ts)
tokens = math.min(capacity, tokens + elapsed * refill)

local allowed = 0
if tokens >= want then
  tokens = tokens - want
  allowed = 1
end

redis.call('HMSET', key, 'tokens', tokens, 'ts', now)
redis.call('EXPIRE', key, math.ceil(capacity / refill) + 60)
return allowed

The Python side is deliberately thin. Every worker constructs one of these, but the object holds no state — the bucket lives entirely under one Redis key:

import time
import redis

class SharedTokenBucket:
    def __init__(self, r: redis.Redis, host: str, rate: float, burst: int):
        self.r = r
        self.key = f"ratelimit:{host}"      # one budget per downstream host
        self.rate = rate
        self.burst = burst
        self._script = r.register_script(LUA)  # LUA = the script above

    def acquire(self, tokens: int = 1) -> bool:
        return bool(self._script(
            keys=[self.key],
            args=[self.burst, self.rate, str(time.time()), tokens],
        ))

Note the key naming: one bucket per downstream host, because budgets are per host. If you later add api.example.com with a different ceiling, it gets its own key and its own rate, and the two never interact.

Now point the reproduction script at it:

r = redis.Redis(host="localhost", port=6379)
bucket = SharedTokenBucket(r, host="store.example.com", rate=1.0, burst=1)

def worker_shared(run_seconds: int):
    pid = os.getpid()
    deadline = time.monotonic() + run_seconds
    with httpx.Client(timeout=10) as client:
        while time.monotonic() < deadline:
            while not bucket.acquire():     # no capacity: wait, don't push
                time.sleep(0.05)
            status = client.get(TARGET).status_code
            print(f"{stamp()} pid={pid} -> {status}", flush=True)

Same four processes. Different world:

14:19:07.312 pid=5301 -> 200
14:19:08.338 pid=5304 -> 200
14:19:09.361 pid=5302 -> 200
14:19:10.389 pid=5301 -> 200

One request per second total. The PIDs rotate because all four workers still participate — the bucket doesn’t care who’s asking, only how much is left.

Two decisions you must make explicitly. First, what happens when Redis is down: fail closed (stop sending) or fail open (send unrestricted). For a hard external budget I want fail closed — a Redis outage should not license a request flood against someone else’s API. Fail open is defensible for internal services with forgiving limits. Second, the polling loop above is simple but chatty; for high-volume callers, compute the sleep duration from remaining tokens instead of polling at 50 ms intervals.

Fix 2: When Redis Is Overkill, Shard the Budget as rate ÷ N per Worker

If your deployment topology is genuinely fixed, there’s a cheaper option: divide the budget statically and give each worker its slice. No new infrastructure, no network hop, no Redis dependency.

# deploy environment
TOTAL_BUDGET=1.0        # req/s that store.example.com tolerates
WORKER_COUNT=4          # must match the actual worker count, exactly

import os

TOTAL_BUDGET = float(os.environ["TOTAL_BUDGET"])
WORKER_COUNT = int(os.environ["WORKER_COUNT"])
RATE_PER_WORKER = TOTAL_BUDGET / WORKER_COUNT   # 0.25 req/s each

# feed RATE_PER_WORKER to whatever limiter you already use

The math works while N holds still:

Workers (N)	Per-worker slice	Combined rate	On budget?
4	0.25 req/s	1.0 req/s	yes
4, one worker idle	0.25 req/s × 3 used	0.75 req/s	yes, but wasting capacity
8 (autoscaler added pods, env stale)	0.25 req/s	2.0 req/s	2x over

Two failure modes hide in that table. Autoscaling is the obvious one: when N changes and nobody recomputes the env var, your budget silently scales with your fleet — the original bug, reintroduced by your autoscaler. The subtle one is utilization: with four slices of 0.25 req/s, a busy worker cannot borrow an idle worker’s headroom. You stay under the cap while leaving throughput on the table, which is a real cost if the downstream tolerates bursts.

My position: static sharding is fine for batch jobs with a pinned worker count and a daily cron schedule. The moment an autoscaler, an orchestrator, or another team’s deployment can change N without a config change flowing through, take the Redis bucket. The extra dependency is cheaper than the incident.

Fix 3: Put the Limiter in Front with a Sidecar Proxy That Owns the Budget

A third option moves enforcement out of your code entirely. Put nginx in front of the downstream host, let it own the budget in a shared-memory zone, and have every worker — every client of any kind — talk to the proxy instead of the target.

# sidecar.conf — one budget in front of all workers
limit_req_zone $binary_remote_addr zone=store_budget:10m rate=1r/s;

server {
    listen 8080;

    location /store/ {
        # rate=1r/s — the sustained budget, common to ALL nginx workers
        # burst=2   — bucket depth: short bursts pass without being refused
        # nodelay   — serve burst requests immediately instead of spacing them out
        limit_req zone=store_budget burst=2 nodelay;
        limit_req_status 429;

        proxy_pass https://store.example.com/;
        proxy_set_header Host store.example.com;
        proxy_ssl_server_name on;
    }
}

Workers now call http://localhost:8080/store/api/orders and their count becomes irrelevant — twenty processes funnel through one zone with one counter. Set limit_req_status explicitly, by the way; the default refusal code is 503, and a 503 masquerading as an origin problem will send you debugging the wrong layer at 2 a.m.

The design detail worth knowing: that memory zone exists precisely because nginx itself runs multiple worker processes. A per-worker counter inside nginx would have the identical multiplication bug you just fixed in Python. The zone keeps one global counter that all nginx workers read and update together. Your bug is old and well-studied; the solution pattern is too.

Trade-offs are real. You’ve added a network hop, TLS termination concerns, Host header plumbing, and one more piece of infrastructure to patch. I’d deploy a sidecar when multiple heterogeneous clients — cron scripts, other services, someone’s notebook — hit the same downstream and you can’t retrofit a common library into all of them. For a single codebase with four workers, the Redis bucket is less machinery to operate.

Stop the Retry Storms: Honor 429s and Retry-After Before They Multiply

A central limiter fixes steady-state behavior. Retries can quietly undo it, because every 429 that triggers an immediate retry adds traffic at the exact moment the server is already refusing you. Four workers that get refused together and retry together have rebuilt the burst you just eliminated.

import random
import time
import httpx

MAX_ATTEMPTS = 5

def fetch_with_backoff(client: httpx.Client, url: str, limiter) -> httpx.Response | None:
    for attempt in range(MAX_ATTEMPTS):
        while not limiter.acquire():      # no capacity: wait, don't pile on
            time.sleep(0.05)

        r = client.get(url)
        if r.status_code != 429:
            return r
        if attempt == MAX_ATTEMPTS - 1:
            return r                      # surface the last 429 to the caller

        # Prefer the server's own number when it provides one.
        retry_after = r.headers.get("Retry-After", "")
        if retry_after.isdigit():
            wait = float(retry_after) + random.uniform(0, 0.5)
        else:
            base = min(2.0 ** attempt, 30.0)
            wait = base + random.uniform(0, base / 2)   # jitter breaks lockstep

        time.sleep(wait)
    return None

Three things in that code earn their keep. The limiter check inside the retry loop means a retry consumes budget like any other request — no bypass lane. The jitter exists because exponential backoff without jitter synchronizes your workers: same schedule, same wakeup, same burst. And Retry-After in integer form is the server telling you its recovery window; ignoring it is leaving free information on the table. Note the header can also carry an HTTP-date rather than seconds — parse that variant with email.utils.parsedate_to_datetime if your target sends it.

The volume difference is not subtle. Model a 10-second outage on the target, four workers, each holding one job that just got refused:

Strategy	Requests sent during the 10s outage	Shape of recovery
Naive: fixed 1s retry loop	~40 (4 workers × ~10 tries)	synchronized retry burst exactly when service returns
Exponential backoff + jitter	~16	staggered ramp
Retry-After aware (header says 5)	~8	retries aligned with the server’s own guidance

That naive row is a self-inflicted denial of service aimed at a recovering server. For a fuller treatment of header handling and backoff ceilings, see your backoff ignoring Retry-After.

One more wrinkle: if your fetching runs through a scraping API rather than raw httpx, the amplification doesn’t disappear — it changes units. FineData’s sync endpoint retries on your behalf with auto_retry enabled by default, and each attempt bills one extra token, so a retry storm lands on your usage report instead of your 429 graph. Centralized retry handling makes storms easier to see, not free to run. The trade-offs between doing retries yourself and delegating them are covered in client-side vs service-side retries.

Verify the Fix: Measure Aggregate Request Rate Before and After

Never declare victory from code review. Load the setup and count what actually escapes toward the target:

# 30 seconds of load through the 4-worker local setup
hey -z 30s -c 8 http://localhost:9000/store/api/orders

One caveat: hey measures what your service accepts, not what leaves toward the downstream. Count on the sending side — at the single choke point where the limiter lives — or on the target’s access logs if you control them. Those counts are what fill this table:

Setup	Requests to store.example.com per 10s	429s per 10s
Broken: 4 per-worker limiters	~40	~30
Redis shared bucket	~10	0
nginx sidecar	~10	0

Then keep the measurement running, because regressions in this bug are silent — nothing errors locally when a new deployment forks another uncoordinated limiter. A counter at the choke point makes the multiplication visible forever:

from prometheus_client import Counter

DOWNSTREAM_REQUESTS = Counter(
    "downstream_requests_total",
    "Requests sent to downstream hosts",
    ["host", "code"],
)

# at the single point every outbound request passes through:
DOWNSTREAM_REQUESTS.labels(host="store.example.com", code=str(r.status_code)).inc()

And one alert rule to page on:

- alert: DownstreamRateBudgetExceeded
  expr: sum by (host) (rate(downstream_requests_total[1m])) > 1

Count at the choke point, not per worker. If each process exports its own metric, you’ve rebuilt the same blind spot in your monitoring that caused the incident.

Wrap-Up

The root cause is structural: in-process limiting state combined with process fan-out. Any limiter constructed inside a worker is a lie the moment worker count exceeds one. Pick your fix by topology — a Redis token bucket for a single codebase that autoscales, static division for pinned batch fleets, an nginx sidecar when many clients share one downstream. Then close the loop: retries that respect the same budget and honor Retry-After, plus an aggregate metric that screams if the rate ever drifts above what the target permits. The bug takes an afternoon to fix and about a minute to verify — as long as you measure the combined rate and not each worker’s conscience.