Throttle MCP Tool Calls: A Token Bucket for Agents

Why Bursts of MCP Tool Calls Break: Quota, Rate Limits, and Stalls

Give an agent an MCP scraping tool and a task like “collect pricing from all 40 products in this catalog,” and watch what happens in the first turn. The model doesn’t issue one call and wait. It fans out. Modern agents will happily fire 30-50 tool invocations in a single response, all of which hit your MCP server simultaneously, and all of which hit the upstream scraping API — or the target site — at once.

Here’s what that looks like in practice. This is a real pattern from an agent run against store.example.com with an unthrottled MCP server:

14:02:11.043  tool=scrape  url=https://store.example.com/products/1   status=200  latency=1.9s
14:02:11.049  tool=scrape  url=https://store.example.com/products/2   status=200  latency=1.8s
14:02:11.051  tool=scrape  url=https://store.example.com/products/3   status=429  retry_after=30
14:02:11.055  tool=scrape  url=https://store.example.com/products/4   status=200  latency=2.1s
...38 more calls within 400ms...
14:02:11.412  tool=scrape  url=https://store.example.com/products/41  status=429  retry_after=30
SUMMARY: 40 calls in 0.37s | 32x 200 | 8x 429 | retry-after penalty: 30s

Forty calls in 370 milliseconds. Eight of them bounced with a 429 and a Retry-After: 30 header. Now the agent either retries all eight at once after sleeping 30 seconds (and possibly trips the limit again), or the run stalls while queued retries pile up behind a single wall. Either way, the wall-clock time for a job that should take ~15 seconds stretches past a minute, and you burned quota on requests that returned nothing usable. If you’re paying per successful page, retries make this worse — a topic covered in detail in How Retries Quietly Double Your Scraping Costs.

There are three ways to pace tool calls, and only one of them behaves well under agent fan-out:

Strategy	Quota burned (40-call burst)	429s hit	Wall-clock time
Unbounded fan-out	40 + ~8 retries	8	~65s (retry wall + backoff)
Naive fixed delay (1s between calls)	40	0	~40s (serializes everything, even when headroom exists)
Token bucket (5 req/s, burst 10)	40	0	~7s (10 immediately, then paced)

The fixed delay is the one people reach for first, and it’s the one I dislike most. It treats every call as equally expensive and serializes your agent even when the target happily accepts 5 requests per second. It’s correct but slow, and it penalizes the fast tools in your server because one slow host dictates the pace of everything. A token bucket gives you the opposite trade: short bursts go through instantly, sustained load gets paced, and the parameters come from measurement rather than guesswork.

Modeling Your Scrape Budget: Measuring Limits Before Writing the Throttle

Do not pick bucket parameters from vibes. The whole point of a token bucket is that refill_rate should approximate the real, observed rate limit of the thing you’re calling — and that number is only findable empirically.

The probe below ramps concurrency against a scraping endpoint, one request per URL, and records the status and Retry-After header for each. It uses the FineData scrape endpoint as the upstream, but the same harness works against any HTTP-fetching MCP tool:

import asyncio
import time
import httpx

API = "https://api.finedata.ai/api/v1/scrape"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}

async def probe_one(client: httpx.AsyncClient, url: str, sem: asyncio.Semaphore):
    async with sem:
        t0 = time.monotonic()
        r = await client.post(API, headers=HEADERS, json={
            "url": url,
            "formats": ["markdown"],
        })
        return {
            "url": url,
            "status": r.status_code,
            "elapsed": round(time.monotonic() - t0, 2),
            "retry_after": r.headers.get("Retry-After"),
        }

async def run_probe(concurrency: int, n: int = 20):
    sem = asyncio.Semaphore(concurrency)
    urls = [f"https://store.example.com/probe/{i}" for i in range(n)]
    async with httpx.AsyncClient(timeout=60) as client:
        results = await asyncio.gather(*(probe_one(client, u, sem) for u in urls))
    ok = sum(1 for r in results if r["status"] == 200)
    limited = [r for r in results if r["status"] == 429]
    print(f"concurrency={concurrency:3d}  ok={ok:3d}  429={len(limited):2d}  "
          f"retry_after={limited[0]['retry_after'] if limited else '-'}")
    return results

for c in (1, 2, 5, 10, 20, 40):
    asyncio.run(run_probe(c))

A typical result looks like this:

Concurrency	200s	429s	Retry-After	Interpretation
1	20	0	—	No constraint at 1 req/s
2	20	0	—	Safe at 2 req/s
5	20	0	—	Safe at 5 req/s
10	20	0	—	Short bursts tolerated
20	16	4	30s	Sustained ceiling is between 5 and 10 req/s
40	12	8	30s	Confirms the ceiling; bursts above ~10 get clipped

From that table you can read off the bucket parameters directly: the sustained rate is ~5 req/s, and bursts up to ~10 get through without penalty. So refill_rate=5 and capacity=10. If you want a safety margin — and you do, because the target’s limit may vary by time of day and your probe ran for 20 seconds, not 20 hours — knock 20% off the refill rate. I’d rather run at 4 req/s forever than eat a 30-second retry wall once an hour.

One caveat: probe from the same environment your agent runs in. If your MCP server routes through residential proxies or a specific proxy_country, the observed ceiling can differ from a bare datacenter probe. Measure what you’ll actually run.

Implementing the Token Bucket: Capacity, Refill Rate, and Fairness

The data structure is simple: a float counter of tokens, a timestamp of the last refill, a capacity, and a refill rate. Tokens accrue at refill_rate per second, capped at capacity. Every caller must acquire one token before proceeding; if none are available, it sleeps until the next one materializes.

import asyncio
import time

class TokenBucket:
    def __init__(self, capacity: float, refill_rate: float):
        self.capacity = float(capacity)       # max tokens held at once
        self.refill_rate = float(refill_rate) # tokens added per second
        self.tokens = float(capacity)         # start full: allow an initial burst
        self.last_refill = time.monotonic()
        self._lock = asyncio.Lock()
        self._cooldown_until = 0.0            # set on 429, see next section

    def _refill(self):
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now

    async def acquire(self):
        """Wait until one token is available. Never blocks the event loop."""
        while True:
            async with self._lock:
                now = time.monotonic()
                if now < self._cooldown_until:
                    # A 429 paused this bucket globally; wait out the cooldown.
                    wait = self._cooldown_until - now
                else:
                    self._refill()
                    if self.tokens >= 1.0:
                        self.tokens -= 1.0
                        return
                    # Seconds until the next full token exists.
                    wait = (1.0 - self.tokens) / self.refill_rate
            await asyncio.sleep(max(wait, 0.01))

    async def penalize(self, seconds: float):
        """Called on a 429: drain the bucket and pause all callers."""
        async with self._lock:
            self.tokens = 0.0
            self._cooldown_until = max(self._cooldown_until,
                                       time.monotonic() + seconds)

Walk through the math for capacity=10, refill_rate=5, and a burst of 25 queued calls:

• Calls 1-10 acquire immediately — the bucket starts full, so the first 10 callers take a token with zero wait. This is the burst allowance.
• Call 11 finds 0 tokens. It computes (1.0 - 0.0) / 5 = 0.2s and sleeps 200ms.
• Call 12 sleeps ~0.2s too, but the lock serializes acquisition: whoever wakes first grabs the token, the next caller recomputes against the remaining fractional tokens. Net effect: calls 11-25 exit at a steady 5 per second.
• Total time for 25 calls: 10 instant + 15 paced at 5/s = 3 seconds. Compare that to 25 serial calls at 1s fixed delay (25 seconds) or 25 simultaneous calls (8+ failures and a 30-second penalty).

Two design details matter more than they look:

The lock is per-acquisition, not per-call. The bucket never holds the lock while a scrape is in flight. It computes the wait, releases the lock, sleeps, and re-contends. This means the event loop stays responsive — other tools on the same MCP server keep answering while 25 scrapes are queued behind the bucket.

Fairness is FIFO-ish by wakeup order. asyncio’s lock is not strictly FIFO, but in practice sleep times are near-identical, so queued calls proceed roughly in submission order. If you need strict ordering (say, a multi-step scrape where step 2 depends on step 1’s session), pass an explicit session_id upstream and don’t rely on queue order.

Wrapping the MCP Tool: A Throttled Tool Wrapper for FastMCP

You could add await bucket.acquire() inside every tool function. Don’t. The moment you have six tools and three buckets, you’ll miss one, and the one you miss is the one the agent will hammer. Wrap instead.

Here’s the raw tool definition, unthrottled:

from fastmcp import FastMCP
import httpx

mcp = FastMCP("scraper")

@mcp.tool()
async def scrape(url: str) -> str:
    """Scrape a URL and return its content as markdown."""
    async with httpx.AsyncClient(timeout=120) as client:
        r = await client.post(
            "https://api.finedata.ai/api/v1/scrape",
            headers={"Authorization": "Bearer fd_your_api_key"},
            json={"url": url, "formats": ["markdown"]},
        )
        r.raise_for_status()
        return r.json()["content"]["markdown"]

And here’s the same server with a throttled wrapper — the tool body is untouched:

import asyncio
import time
from functools import wraps
from urllib.parse import urlparse

from fastmcp import FastMCP
import httpx

mcp = FastMCP("scraper")

# One bucket per host, parameters from the probe results.
BUCKETS = {
    "store.example.com": TokenBucket(capacity=10, refill_rate=5),
    "example.com":      TokenBucket(capacity=4,  refill_rate=2),
}
DEFAULT_BUCKET = TokenBucket(capacity=6, refill_rate=3)

def bucket_for(url: str) -> TokenBucket:
    host = urlparse(url).hostname or ""
    return BUCKETS.get(host, DEFAULT_BUCKET)

def throttled(tool_fn):
    """Wrap an MCP tool so every call passes through the host's bucket."""
    @wraps(tool_fn)
    async def wrapper(url: str, *args, **kwargs):
        bucket = bucket_for(url)
        await bucket.acquire()          # pace BEFORE the request, not after
        try:
            return await tool_fn(url, *args, **kwargs)
        except httpx.HTTPStatusError as e:
            if e.response.status_code == 429:
                retry_after = float(e.response.headers.get("Retry-After", 30))
                await bucket.penalize(retry_after)
            raise
    return wrapper

@mcp.tool()
@throttled
async def scrape(url: str) -> str:
    """Scrape a URL and return its content as markdown."""
    async with httpx.AsyncClient(timeout=120) as client:
        r = await client.post(
            "https://api.finedata.ai/api/v1/scrape",
            headers={"Authorization": "Bearer fd_your_api_key"},
            json={"url": url, "formats": ["markdown"]},
        )
        r.raise_for_status()
        return r.json()["content"]["markdown"]

The diff is three lines of intent: @throttled on the tool, acquire() before the request, and penalize() on a 429. The tool implementation stays ignorant of pacing, which means you can bolt this onto an existing server without touching tool logic.

One subtlety: the wrapper keys the bucket on the destination host, parsed from the url argument. That’s the right default because rate limits are per-target, not per-tool. If you also have an account-level quota on the scraping API itself (say, N requests per minute across all targets), add a second, global bucket and acquire from both. Nested acquisition is fine as long as both acquire() calls are awaited sequentially — deadlock isn’t possible here because nothing holds a token while waiting for another.

Bursts vs. Sustained Load: Tuning Bucket Parameters per Tool and per Host

Per-host buckets solve the starvation problem that a single global throttle creates. If example.com tolerates 2 req/s and store.example.com tolerates 5 req/s, a shared bucket forces everything down to the slowest host. Separate buckets let each target run at its own ceiling, and a crawl of the slow host doesn’t block the agent’s one-off lookup against the fast one.

Move the parameters into config so you can tune without redeploying:

throttle:
  default:
    capacity: 6
    refill_rate: 3
  hosts:
    store.example.com:
      capacity: 10
      refill_rate: 5
    example.com:
      capacity: 4
      refill_rate: 2

Now the part people argue about: burst capacity. Capacity is not a safety parameter — it’s a latency trade-off. Small capacity means smooth pacing but a slow start on every batch. Large capacity means the first N calls of any fan-out land instantly, but a batch that arrives right after another batch has drained the bucket waits longer at the front.

Measured on a 50-page crawl of store.example.com (sustained ceiling 5 req/s, so total time has a hard floor of ~8 seconds):

Parameter set	Capacity / refill	First 10 calls	Total crawl time	429s
Small burst	2 / 5 per s	0.4s	9.6s	0
Balanced	10 / 5 per s	instant	8.1s	0
Large burst	25 / 5 per s	instant	8.0s	0

The large burst buys you 0.1 seconds and costs you the ability to absorb a second concurrent agent’s fan-out without tripping the limit. The small burst costs 1.5 seconds on every single crawl, forever. Balanced wins: capacity roughly equal to “what the target tolerates in one instant” per your probe, refill equal to the sustained ceiling minus a safety margin.

My opinionated take: set capacity to the burst size the target demonstrably tolerates, and no higher. Capacity above the observed burst tolerance is pure downside — it converts headroom you don’t have into 429s you’ll pay for. If you run multiple agents against the same MCP server, they share the bucket, which is exactly what you want; the bucket is the coordination point that the agents themselves can’t provide. This is also why I’d put the throttle in the MCP server rather than in the agent’s client code — a server-side bucket protects you from every client, including the ones you didn’t write. The same reasoning applies to tracing agent fetches, which Debug MCP Tool Calls covers from the observability side.

Handling 429s Gracefully: Retry-After-Aware Backoff Inside the Throttle

Even a well-tuned bucket leaks occasionally. The target’s limit might tighten mid-run, or another tenant of the same scraping API might be consuming shared capacity. The failure mode to avoid is thundering retries: 15 queued calls all hit the wall, all sleep 30 seconds, all retry at the same instant, and hit the wall again.

The penalize() method in the bucket handles this. On a 429, it drains the bucket to zero and sets a cooldown timestamp. Every queued caller — not just the one that got the 429 — waits out the cooldown, because acquire() checks _cooldown_until before touching tokens. One failure pauses the whole pipeline; the pipeline then resumes at the normal refill rate instead of all at once.

Wire the Retry-After header through so the cooldown honors what the server actually asked for:

def parse_retry_after(response) -> float:
    raw = response.headers.get("Retry-After", "30")
    try:
        return float(raw)
    except ValueError:
        # HTTP-date format: fall back to a conservative default.
        return 30.0

@mcp.tool()
@throttled
async def scrape(url: str) -> str:
    """Scrape a URL and return its content as markdown."""
    bucket = bucket_for(url)  # exposed to the tool via a closure or context
    async with httpx.AsyncClient(timeout=120) as client:
        r = await client.post(
            "https://api.finedata.ai/api/v1/scrape",
            headers={"Authorization": "Bearer fd_your_api_key"},
            json={"url": url, "formats": ["markdown"]},
        )
        if r.status_code == 429:
            await bucket.penalize(parse_retry_after(r))
            r.raise_for_status()
        r.raise_for_status()
        return r.json()["content"]["markdown"]

In a real run it looks like this — one 429, then silence:

14:07:02.110  scrape store.example.com/products/23  429  retry_after=15
14:07:02.111  bucket[store.example.com] penalized: tokens=0 cooldown=15s
14:07:17.2xx  scrape store.example.com/products/23  200  (re-acquired after cooldown)
14:07:17.4xx  scrape store.example.com/products/24  200
...remaining 40 calls paced at 5/s...
14:07:25.9xx  SUMMARY: 50 calls | 1x 429 | 0 429s after cooldown | total 24s

One 429, a 15-second global pause, and then forty subsequent calls with zero additional rate-limit hits. Compare that to the unthrottled run at the top of this post: eight 429s in the first 400ms, then a second wave of failures after the retry wall.

A word of caution: don’t put auto_retry-style service-side retries and your own throttle-side retries in the same path without accounting for both. If the upstream scraping call retries internally, your bucket sees one acquire but the target sees several requests. Either disable the upstream retry and own the retry logic in the wrapper, or set the bucket’s refill rate low enough to absorb the multiplier. Mixing both without adjusting is how people end up throttled at 1 req/s while convinced they’re running at 5.

Verifying the Throttle: Load-Test Your Agent and Watch the Pacing Curve

A throttle you haven’t measured is a hope, not a mechanism. This harness simulates the worst realistic case — an agent issuing 60 concurrent tool calls in a single turn — and timestamps every request:

import asyncio
import time
from collections import deque

async def simulated_agent_call(bucket, i, log: deque):
    await bucket.acquire()
    log.append((time.monotonic(), i))

async def main():
    bucket = TokenBucket(capacity=10, refill_rate=5)
    log = deque()
    t0 = time.monotonic()

    # 60 "tool calls" fired simultaneously, like a single agent turn.
    await asyncio.gather(*(simulated_agent_call(bucket, i, log)
                           for i in range(60)))

    total = time.monotonic() - t0
    print(f"60 calls in {total:.2f}s (theoretical floor: {50/5:.1f}s)")

    # Emit timestamps for plotting.
    with open("pacing.csv", "w") as f:
        f.write("t,call\n")
        for t, i in log:
            f.write(f"{t - t0:.3f},{i}\n")

asyncio.run(main())

The expected output: 60 calls in ~10.0s. The first 10 calls complete within milliseconds of t=0 — that’s the burst capacity draining. The remaining 50 proceed at 5 per second. If total time is significantly above 10s, your acquire() has a bug (usually a lock held too long or a sleep computed against stale token state). If the first 10 calls are not instant, the bucket started empty — check that __init__ sets tokens = capacity.

To generate the pacing chart, load pacing.csv into any plotting tool and graph cumulative calls against time. The curve you want looks like this: a near-vertical jump to 10 within the first 50ms, then a straight line with a slope of 5 calls per second out to 60. A fixed-delay throttle produces a slope-1 line from the origin — correct but 4x slower for this workload. An unthrottled run produces a vertical jump to 60 with a scatter of failures on top.

Run this test against the real MCP server too, not just the bucket in isolation. Fire 60 actual tool invocations through the server (a small script using the MCP client SDK works fine) and confirm the server’s response timestamps match the bucket’s prediction. The wrapper is three lines of code; the failure modes — a decorator applied in the wrong order, a bucket keyed on the wrong argument, a second tool added without @throttled — all live in the wiring, not the bucket.

Gotchas and Next Steps

Bucket keying. Key on destination host, not on tool name. Two tools hitting the same host share one limit whether you like it or not; two tools hitting different hosts should never share a bucket. If a tool takes multiple URLs in one call, acquire one token per URL before issuing anything.

Don’t throttle local tools. A search_notes tool on the same server doesn’t need a bucket, and giving it one adds latency for nothing. Wrap only tools that perform network fetches.

Timeout interaction. If your tool has a 120-second upstream timeout and the queue behind the bucket is 40 deep, the effective wait for the last caller is 8 seconds of pacing plus scrape time. Make sure the client-side timeout on the MCP call itself exceeds pacing wait plus scrape latency, or the agent will cancel calls that were about to succeed.

Prometheus-style counters. Export tokens_consumed, wait_seconds_p95, and penalties per bucket. A creeping p95 wait is your early signal that a target tightened its limit and your refill rate is now too optimistic.

Next steps worth the time: extend the probe to measure limits per hour of the day and adjust refill rates on a schedule; add a second global bucket if your scraping API has account-level quota; and read Async Scraping at Scale: Jobs, Batches, Webhooks if your workloads are large enough that submitting batches and pacing the submission rate beats pacing individual calls. For most agent workloads, though, the token bucket above is the whole fix: measured parameters, per-host buckets, one global pause on 429, and a harness to prove it works.