Sync vs Async Scraping: When Blocking Requests Are Enough

Async Isn’t Free Speed: The Trade Nobody Quotes

Every scraping codebase I’ve inherited has the same skeleton in the closet: someone rewrote a perfectly good requests loop with asyncio, got a 15x speedup in the dev environment, and then shipped it into production where it tripped the target’s rate limiter within an hour. Async is not free speed. It is a trade between wall-clock time and about six other things you care about: ban rate, debuggability, retry semantics, cost per successful response, and how much of your weekend the on-call rotation loses.

This post compares the two styles honestly. Sometimes blocking code wins. I’ll show you exactly when, with arithmetic you can reproduce.

The Mental Model: One Worker, One Request vs. a Pool of Coroutines

Before any performance numbers make sense, you need the right picture of what the machine is actually doing.

Synchronous code is one worker with one in-tray. The worker picks up a request, sends it, and then sits there — blocked on the socket — until the bytes come back. The CPU is essentially idle during that wait. Here’s the loop everyone has written:

import requests

urls = [
    "https://store.example.com/products/1",
    "https://store.example.com/products/2",
    "https://store.example.com/products/3",
]

for url in urls:
    resp = requests.get(url, timeout=10)   # <-- thread parks here, doing nothing
    print(resp.status_code, len(resp.content))

Between requests.get() sending the request and the response arriving, that thread does nothing useful. Multiply by 500 URLs and 800ms average latency, and you’ve built a machine that converts 99% of its runtime into waiting.

The asyncio version looks almost identical, which is part of the problem — the difference is invisible in the source and enormous at runtime:

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as resp:
        # the await below is where this coroutine SUSPENDS and
        # hands control back to the event loop
        body = await resp.read()
        print(resp.status, len(body))

async def main():
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[fetch(session, u) for u in urls])

asyncio.run(main())

The await is the whole trick. At that point the coroutine says “I’m waiting on the network; run someone else,” and the event loop switches to another fetch. A single thread interleaves dozens of in-flight requests. The timeline looks like this:

SYNC (one worker, serialized waits)

req1  |--send--IDLE IDLE IDLE--recv--|
req2                                 |--send--IDLE IDLE IDLE--recv--|
req3                                                                  |--send--IDLE--recv--|

ASYNC (one thread, interleaved coroutines)

req1  |--send--IDLE~IDLE~IDLE~IDLE~IDLE~recv--|
req2  |  |--send--IDLE~IDLE~IDLE~IDLE~recv--| |
req3  |  |  |--send--IDLE~IDLE~IDLE~recv--| | |
      ~ = event loop running other coroutines during this wait

Same thread. Same CPU. The idle gaps get filled with other requests’ work. That’s the entire mechanism — no parallelism, just cooperative concurrency over blocking I/O.

One consequence that bites people: if any code inside the coroutine does CPU-heavy work or calls a blocking library (a sync database driver, time.sleep, a heavy parser), the whole loop stalls. Every other coroutine waits for it. Sync code is forgiving of this; async code punishes it silently.

Throughput Math: Why Async Speeds Up Only When the Server Makes You Wait

The speedup formula is trivially simple, and nobody quotes it:

sync_rpm  = 60 / latency_seconds
async_rpm = (60 / latency_seconds) * concurrency

Async multiplies throughput by your concurrency factor, but only if latency is real. If the server responds in 5ms, there’s almost nothing to fill and fan-out buys you nothing. If it responds in 2s, fan-out buys you a lot. Here’s the table:

Avg response time	Concurrency	Sync RPM	Async RPM
100ms	1	600	600
100ms	10	600	6,000
100ms	50	600	30,000
500ms	1	120	120
500ms	10	120	1,200
500ms	50	120	6,000
2s	1	30	30
2s	10	30	300
2s	50	30	1,500

Worked example: 500 product URLs off store.example.com, 800ms average latency.

• Sync: 500 × 0.8s = 400 seconds (~6.7 minutes).
• Async, concurrency 20: 500 / 20 = 25 waves × 0.8s = 20 seconds.

A 20x speedup, exactly matching the concurrency factor. The arithmetic is seductive. Here’s where it breaks down in practice:

1. The 800ms is not constant. At concurrency 20, the target server queues you. Some requests take 800ms; the stragglers take 4s. Your effective latency rises with your concurrency, so the speedup is sublinear.
2. Connection setup costs. Sync with a requests.Session reuses one keep-alive connection. Async opens many, paying TLS handshakes and hitting ephemeral port limits on the client side (check ulimit -n before you blame the server).
3. The target notices. 1,500 RPM from one IP is not a traffic pattern; it’s an attack signature. Which brings us to the next section.

The honest summary: async’s advantage is proportional to how much the server makes you wait, and inversely proportional to how much the server is willing to tolerate you.

Rate Limits Turn Async Speed Into Async Bans: Throttling Both Approaches Fairly

The throughput table above is a fiction in one important case: any target with a rate limit. If store.example.com serves 10 requests per second per IP before returning 429, then your ceiling is 600 RPM regardless of style. Async doesn’t beat the limit — it just reaches it faster and then slams into it harder.

So both styles need throttling, and once both are throttled to the same request rate, their throughput converges. The remaining difference is structural: sync throttles naturally (one in-flight request at a time), while async throttles by explicit constraint.

Sync with a fixed delay — simple, predictable, easy to reason about:

import requests
import time

session = requests.Session()
DELAY = 0.1  # 10 req/s ceiling, comfortably under the documented limit

for url in urls:
    resp = session.get(url, timeout=10)
    handle(resp)
    time.sleep(DELAY)   # politeness is structural: nothing else can fire early

Async with a semaphore plus jitter. The semaphore caps concurrency; the jitter keeps you from sending perfectly spaced bursts that look like a robot:

import aiohttp
import asyncio
import random

MAX_CONCURRENCY = 10

async def fetch(session, sem, url):
    async with sem:                       # caps in-flight requests at 10
        # jitter: without this, all 10 slots fire in lockstep every
        # ~800ms, producing a sawtooth pattern that rate limiters flag
        await asyncio.sleep(random.uniform(0.0, 0.15))
        async with session.get(url) as resp:
            return resp.status, await resp.read()

async def main():
    sem = asyncio.Semaphore(MAX_CONCURRENCY)
    async with aiohttp.ClientSession() as session:
        await asyncio.gather(*[fetch(session, sem, u) for u in urls])

asyncio.run(main())

Remove the semaphore and the jitter, and what happens? All 500 requests enter the event loop immediately. aiohttp’s default connector allows 100 concurrent connections, so you send a hundred simultaneous requests, get a wall of 429s, and — if you naively retry — re-trigger the limit in a tight loop. Async amplifies both your speed and your mistakes.

Comparing the two throttled approaches:

Metric	Sync (fixed delay)	Async (semaphore + jitter)
Requests per window	Deterministic, exactly the delay rate	Bounded by semaphore, slightly bursty
429 probability	Near zero if delay is set correctly	Low, but bursty edges can trip strict limiters
Politeness overhead	The delay is the runtime	Semaphore costs nothing when slots are free
Time to adjust rate	Edit one constant	Edit semaphore value, reason about interaction with timeouts

My opinion, which some will disagree with: for a single target with a strict, documented rate limit, sync is the better tool. The delay loop is self-documenting and cannot exceed its own budget. Async’s advantage — filling idle time — is precisely what the rate limit forbids you from doing. If you’re going to throttle async down to sync speed anyway, you’ve taken on coroutine complexity for nothing. Fan-out pays when you have many independent targets, not one throttled one. That’s the pattern behind async scraping jobs and webhooks, where a batch of URLs fans out across a fleet instead of hammering one host.

Debugging and Stack Traces: Why Blocking Code Fails Loudly and Coroutines Fail Confusingly

This section is the one performance articles skip, and it’s the one that decides your 3 a.m. incident length.

Sync failure is legible. One request, one traceback, one URL:

import requests

try:
    resp = requests.get("https://store.example.com/products/42", timeout=10)
    resp.raise_for_status()
except requests.Timeout as e:
    # full, linear traceback: line, request, URL, elapsed time
    print(f"TIMEOUT: {e.request.url}")
except requests.HTTPError as e:
    print(f"HTTP {e.response.status_code}: {e.request.url}")

Set a breakpoint on the except. Inspect e.request, the response, the URL. Reproduce it with a single curl. Done. Blocking code fails at the speed of human comprehension.

Async failure is a different animal. Here’s the broken version:

results = await asyncio.gather(*[fetch(session, u) for u in urls])
# one exception in ANY task cancels the others and propagates
# with a traceback pointing at gather(), not at the failing URL

gather() without return_exceptions=True has two nasty behaviors: the first exception cancels all sibling tasks (your in-flight work evaporates), and the traceback points at the gather line, not the request. You know something failed. You don’t know what, or how far the others got. The fixed version:

async def fetch_and_log(session, url):
    try:
        async with session.get(url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
            return url, resp.status, await resp.read()
    except Exception as e:
        # per-task logging: the exception is attributed to its URL
        print(f"FAILED {url}: {type(e).__name__}: {e}")
        return url, None, None

# exceptions never escape; you get a full result set with failures marked
results = await asyncio.gather(
    *[fetch_and_log(session, u) for u in urls],
    return_exceptions=True,
)

Feature-by-feature:

• Breakpoints: sync stops the world and shows you everything. Async breakpoints inside a coroutine freeze the loop but 49 sibling coroutines are mid-flight in a state you can’t easily inspect.
• Reproducibility: sync failure reproduces with one request. Async failure often depends on interleaving — a timeout that appears only when 50 other requests are competing for the loop. Heisenbugs.
• Error-to-URL mapping: sync gives it to you for free. Async gives you a list of (url, None, None) and makes you write the attribution logic yourself.

If your team’s debugging playbook is “reproduce, isolate, fix,” async adds a fourth step before the first: figure out which of the 500 interleaved tasks actually broke. Budget for it.

Cost per Successful Request: Counting Retries, Proxies, and Burned Bandwidth

Raw throughput is the wrong metric. What you actually pay for is successful responses. A 429 costs you a request, a retry, and possibly a burned proxy IP, and delivers zero data. Let’s make that concrete with a small cost model:

def cost_per_success(total_requests, ok, retries, proxy_cost_per_req, token_cost):
    """Compute effective cost per 200-OK from raw crawl metrics."""
    wasted = total_requests - ok          # 4xx/5xx/timeout: paid for, no data
    spend = total_requests * (proxy_cost_per_req + token_cost)
    # retries double-pay: the retry itself is another request
    spend += retries * (proxy_cost_per_req + token_cost)
    return spend / ok

# store.example.com crawl, lenient target
sync_cost  = cost_per_success(total=1050, ok=1000, retries=50,
                              proxy_cost_per_req=0.001, token_cost=0.002)
async_cost = cost_per_success(total=1220, ok=1000, retries=220,
                              proxy_cost_per_req=0.001, token_cost=0.002)

print(f"sync:  ${sync_cost:.5f} per success")
print(f"async: ${async_cost:.5f} per success")

The point of the model, not the specific prices: every failed request is money spent for nothing, and retries are that money spent twice. Now the comparison that matters:

Scenario	Style	Requests sent	Failures (429/timeout)	Retries	Cost per success
Lenient target	Sync, low concurrency	1,050	50	50	$0.00315
Lenient target	Async, high concurrency	1,220	220	220	$0.00366
Strict rate limiter	Sync, throttled	1,020	20	20	$0.00306
Strict rate limiter	Async, throttled to same rate	1,020	20	20	$0.00306

Read that last row carefully. Once you throttle async to the rate the strict target allows, its cost per success is identical to sync — because identical requests are being sent at an identical rate. The only remaining differences are wall-clock time (async finishes its batch no faster, since the rate caps both) and code complexity (async is strictly worse).

Against a lenient target, async’s higher concurrency produces more failures — the server degrades under load, stragglers time out, and the retry loop compounds it. The cost per success rises even though the crawl finished faster. Whether that trade is worth it depends on whether your bottleneck is time or budget, and most scraping budgets I’ve seen are tighter than their deadlines.

There’s a hidden cost the table can’t show: IP burn. A 429 is a warning shot. Escalating to bans means rotating proxies earlier than planned, and proxy rotation strategies have their own per-IP warmup and reputation costs. Aggressive concurrency spends IP reputation — a resource that’s expensive to replenish and that no per-request token price captures.

Local Measurement: Benchmarking Both Styles Against a Test Server Before You Commit

Rules of thumb are a poor substitute for measuring your own workload. The crossover point between sync and async depends entirely on your latency profile, so build a tiny test server and find it empirically.

First, a local server with configurable delay:

# server.py -- local target with simulated latency
from aiohttp import web
import asyncio

DELAY = 0.2  # 200ms per request

async def item(request):
    await asyncio.sleep(DELAY)
    return web.json_response({"id": request.match_info["n"], "price": 19.99})

app = web.Application()
app.router.add_get("/item/{n}", item)
web.run_app(app, port=8899)

Then the benchmark harness, running both styles over the same 200 URLs:

# bench.py
import asyncio, time, aiohttp, requests
from statistics import median

URLS = [f"http://127.0.0.1:8899/item/{i}" for i in range(200)]

def bench_sync():
    t0 = time.perf_counter()
    latencies, ok = [], 0
    s = requests.Session()
    for u in URLS:
        t = time.perf_counter()
        r = s.get(u, timeout=10)
        latencies.append(time.perf_counter() - t)
        ok += r.status_code == 200
    return time.perf_counter() - t0, ok, sorted(latencies)[int(len(latencies)*0.95)]

async def bench_async(concurrency=20):
    t0 = time.perf_counter()
    sem, latencies, ok = asyncio.Semaphore(concurrency), [], 0
    async def one(s, u):
        nonlocal ok
        async with sem:
            t = time.perf_counter()
            async with s.get(u) as r:
                await r.read()
            latencies.append(time.perf_counter() - t)
            ok += r.status_code == 200
    async with aiohttp.ClientSession() as s:
        await asyncio.gather(*[one(s, u) for u in URLS])
    return time.perf_counter() - t0, ok, sorted(latencies)[int(len(latencies)*0.95)]

wall, ok, p95 = bench_sync()
print(f"SYNC  : {wall:.2f}s  ok={ok}/200  p95={p95*1000:.0f}ms")
wall, ok, p95 = asyncio.run(bench_async())
print(f"ASYNC : {wall:.2f}s  ok={ok}/200  p95={p95*1000:.0f}ms")

Typical output on my machine, 200ms simulated latency, 200 URLs:

SYNC  : 41.02s  ok=200/200  p95=205ms
ASYNC :  2.31s  ok=200/200  p95=214ms

Interpretation: at 200ms latency, the crossover is immediate — async with concurrency 20 is ~18x faster, and p95 barely moves because the local server doesn’t degrade. Now change DELAY to 0.02 and rerun. Sync takes ~4.5s; async takes ~0.4s. Still faster, but the gap compressed from 18x to 11x, and the absolute savings shrank from 39 seconds to 4. Keep lowering latency and you reach the point where the event loop’s own overhead is a visible fraction of the runtime. That’s your workload’s crossover, and no blog post can tell you where it sits.

The benchmark also gives you a safe place to tune your semaphore value. Raise concurrency until p95 latency starts climbing — that’s the server (or your client) saturating, and it’s the same signal you’ll see against a real target right before the 429s start.

Decision Table: Choosing Sync or Async by Workload Shape, Not by Fashion

Everything above compresses to this:

Scenario	Recommended style	Key reason	Concurrency setting
Small one-off crawl (< a few hundred URLs)	Sync	Total runtime is minutes either way; debuggability wins	1
Latency-heavy target (1s+ responses, lenient limits)	Async	Latency is the dominant cost; fan-out fills idle time	10–20, tune to p95
Strict documented rate limit	Sync	Throttled async converges to sync throughput with added complexity	1 + fixed delay
Overnight batch, millions of URLs	Async	Wall-clock time is the binding constraint; failures are cheap to re-run	50+, with backoff
Interactive tool / script run by humans	Sync	Predictability and clear errors beat speed	1
Many independent targets, one job each	Async	Concurrency across hosts rarely trips per-host limits	20–50

And the pattern that covers most real projects: hybrid. Sync orchestration — pagination logic, checkpointing, retries, database writes — wrapping an async fan-out for the inner loop. Here’s the sketch, fetching a category listing first, then fanning out over the product URLs it contains:

import asyncio, aiohttp, requests

def get_product_urls(category_url):
    # sync: one blocking call, easy to debug, easy to checkpoint
    html = requests.get(category_url, timeout=10).text
    return parse_product_links(html)   # -> list of store.example.com URLs

async def fetch_all(urls, concurrency=15):
    sem = asyncio.Semaphore(concurrency)
    async def one(s, u):
        async with sem:
            async with s.get(u) as r:
                return u, r.status, await r.text()
    async with aiohttp.ClientSession() as s:
        return await asyncio.gather(
            *[one(s, u) for u in urls], return_exceptions=True)

for page in range(1, 50):
    urls = get_product_urls(f"https://store.example.com/c/gadgets?page={page}")
    results = asyncio.run(fetch_all(urls))   # sync code waits, then continues
    save_to_db(results)

The pagination loop stays linear and debuggable; the expensive inner loop fans out. If a page fails, the traceback points at the pagination line, not into a coroutine pile-up. The same shape applies if you offload the fan-out to a hosted batch API instead of running it in-process — submit the URL list, poll or take a webhook, keep your orchestration blocking and boring, which is exactly what you want orchestration to be.

Where This Leaves You

Sync and async are not competing philosophies. They’re tools whose value depends on one variable: how much the target makes you wait, and how much waiting it will tolerate.

Blocking code wins on debuggability, error attribution, natural politeness, and any workload where a rate limit caps throughput anyway. Async wins on wall-clock time for latency-heavy, limit-lenient targets, and you pay for it with semaphore tuning, jitter, per-task error handling, and interleaving-dependent failures. Measure your own crossover with a local delay server before committing — the arithmetic is simple, but the constants are yours.

If you take one rule from this: never adopt async for its own sake, and never throttle async down to sync speed. Either commit to the fan-out or don’t do it. Half-throttled coroutines give you the worst of both worlds, at the highest maintenance cost.