Skip to content
Technical 14 min read

Prove Your Concurrency: Count In-Flight Requests Locally

Point your scraper at a local listener that logs request starts and ends, then compute real in-flight concurrency to verify your worker pool setup.

FE
FineData Engineering · Editorial Policy
|
On this page

Why Configured Workers Don’t Equal Measured Concurrency

You set workers=20 in your scraper config, the logs show twenty threads spinning, and throughput is exactly what it was at workers=1. The config says concurrent. The wire says serial.

This happens constantly, and it happens for reasons that never show up in your own logs:

  • A shared requests.Session behind a lock, so every fetch queues on the same mutex
  • A single connection pool capped at one connection per host
  • Retry logic with a global backoff gate that serializes everything after the first 429
  • A DNS resolution step that blocks the whole event loop in an “async” scraper

Your scraper’s own metrics can’t catch this, because from the inside every worker looks busy. The only way to know is to watch the requests arrive from the outside — and the easiest place to watch from is a listener you control, on your own machine.

Here’s the pattern I use on every new crawler before it ever touches a real target. Point the scraper at a local HTTP listener that timestamps every request start and end, then compute the actual in-flight concurrency from those events. Five minutes of setup, and it settles the argument with numbers.

To make the failure concrete, here’s roughly what the gap looks like in practice:

ConfigurationConfigured workersMeasured peak in-flightWall time (50 URLs)
workers=11148s
workers=5 (shared session lock)5147s
workers=20 (shared session lock)20244s

Twenty workers, two requests on the wire. The rest are parked on a lock. That’s the bug we’re going to catch.

And here’s the kind of code that produces it — a worker loop that looks concurrent:

import threading
import requests

session = requests.Session()
session_lock = threading.Lock()   # the culprit

def fetch(url):
    with session_lock:            # only one worker can fetch at a time
        resp = session.get(url, timeout=10)
    return resp.text

def worker(queue):
    while True:
        url = queue.get()
        if url is None:
            return
        fetch(url)
        queue.task_done()

Nothing here raises an error. Every request succeeds. The logs print one line per URL with no delay you’d notice. But the lock means the pool is a queue with extra steps. If you want the same measurement against a managed scraping API instead of a raw target, the same technique applies — submit N async jobs, count how many report processing simultaneously, and compare that to your intended parallelism. The local listener version below is just cheaper to iterate on.

Build a Local Listener That Timestamps Request Starts and Ends

The listener’s only job is to write two events per request: one when the request arrives, one when the handler finishes. JSONL is the right format — append-only, one event per line, trivially parseable later.

# listener.py
import json
import time
import argparse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

LOG_PATH = "requests.jsonl"

def log_event(event, path):
    with open(LOG_PATH, "a") as f:
        f.write(json.dumps({
            "event": event,
            "ts": time.time(),
            "path": path,
        }) + "\n")

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        log_event("start", self.path)
        # simulate a slow origin: 800ms response time
        time.sleep(0.8)
        body = b'{"ok": true}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
        log_event("end", self.path)

    def log_message(self, *args):
        pass  # silence default stderr logging

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--port", type=int, default=8080)
    parser.add_argument("--delay", type=float, default=0.8)
    args, _ = parser.parse_known_args()
    Handler.delay = args.delay

    server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
    print(f"listening on 127.0.0.1:{args.port}, delay={args.delay}s")
    server.serve_forever()

Two details matter. First, ThreadingHTTPServer — the single-threaded variant serializes requests itself and would poison your measurement. Second, the time.sleep stands in for real network latency; without it, requests complete so fast that even a serial scraper looks bursty. A fixed 800ms delay makes overlap unambiguous: if your scraper is genuinely concurrent, start events should arrive nearly simultaneously.

Run it and fire five synthetic requests at it:

python listener.py &
for i in 1 2 3 4 5; do
  curl -s "http://127.0.0.1:8080/products/$i" > /dev/null &
done
wait

The log looks like this:

{"event": "start", "ts": 1730000000.112, "path": "/products/1"}
{"event": "start", "ts": 1730000000.114, "path": "/products/2"}
{"event": "start", "ts": 1730000000.117, "path": "/products/3"}
{"event": "start", "ts": 1730000000.119, "path": "/products/4"}
{"event": "start", "ts": 1730000000.121, "path": "/products/5"}
{"event": "end", "ts": 1730000000.913, "path": "/products/1"}
{"event": "end", "ts": 1730000000.915, "path": "/products/2"}
{"event": "end", "ts": 1730000000.918, "path": "/products/3"}
{"event": "end", "ts": 1730000000.920, "path": "/products/4"}
{"event": "end", "ts": 1730000000.922, "path": "/products/5"}

Five starts, then five ends, roughly 800ms apart. That’s what true concurrency looks like in this log. A serial scraper produces start/end pairs strictly interleaved: start, end, start, end. You can read the bug off the page.

One caveat: time.time() is fine for a single-process listener, but if you ever split the listener across processes, switch to a monotonic shared clock or log in one process only. Wall-clock skew between processes will corrupt the sweep later.

Rewrite Your Scraper’s store.example.com URLs to the Local Listener

You should not have to edit your crawler’s logic to test it. If your fetch function builds URLs from a base, make the base a configuration value.

# config.py
import os

# Point this at the local listener to measure concurrency.
# Any real run overrides it via the environment.
BASE_URL = os.environ.get("SCRAPER_BASE_URL", "https://store.example.com")
# real run
python crawl.py

# measurement run
SCRAPER_BASE_URL="http://127.0.0.1:8080" python crawl.py

The rewrite in the fetch function is a three-line diff. Preserve paths and query strings exactly — pagination cursors, category filters, everything:

 def build_url(path, params=None):
-    return f"https://store.example.com{path}"
+    base = BASE_URL
+    if params:
+        return f"{base}{path}?{urlencode(params)}"
+    return f"{base}{path}"

If your scraper hardcodes https://store.example.com in twenty places, that’s your first finding: the crawl isn’t redirectable, which also means you can’t point it at a staging mirror, a proxy, or a scraping API endpoint without a refactor. Fix that before anything else.

One thing to watch: TLS. The real target is HTTPS; the listener is plain HTTP. Most HTTP clients handle the scheme from the URL string, so this is usually a non-issue — but if your client pins a TLS context or you’re testing a client that validates certificates against a fixed hostname, run the listener behind a local reverse proxy with a self-signed cert instead. The measurement technique is identical; only the listener setup gets heavier.

Compute In-Flight Concurrency from Start/End Event Pairs

Now the actual math. Every request contributes two events: (+1, start_ts) and (-1, end_ts). Sort all events by timestamp, sweep through them, and track the running count. The count after each event is the number of requests in flight at that instant.

# analyze.py
import json
import sys
from collections import defaultdict

def load_events(path):
    events = []
    with open(path) as f:
        for line in f:
            rec = json.loads(line)
            delta = 1 if rec["event"] == "start" else -1
            events.append((rec["ts"], delta, rec["path"]))
    # sort by ts; on ties, process starts before ends
    # so a request ending at the same instant one starts counts as overlap-free
    events.sort(key=lambda e: (e[0], e[1]))
    return events

def sweep(events):
    count = 0
    peak = 0
    peak_paths = []
    samples = []  # (ts, count) after each event
    for ts, delta, path in events:
        count += delta
        samples.append((ts, count))
        if count > peak:
            peak = count
    return peak, samples

def total_inflight_time(samples):
    # area under the count curve, in seconds
    area = 0.0
    for (t0, c), (t1, _) in zip(samples, samples[1:]):
        area += (t1 - t0) * c
    return area

events = load_events(sys.argv[1] if len(sys.argv) > 1 else "requests.jsonl")
peak, samples = sweep(events)

span = samples[-1][0] - samples[0][0] if samples else 0
avg = total_inflight_time(samples) / span if span > 0 else 0

print(f"requests:        {sum(1 for e in events if e[1] == 1)}")
print(f"max in-flight:   {peak}")
print(f"avg in-flight:   {avg:.2f}")
print(f"wall-clock span: {span:.2f}s")

For the five-request curl burst above, this prints:

requests:        5
max in-flight:   5
avg in-flight:   4.98
wall-clock span: 0.81s

Average in-flight of 4.98 against a peak of 5 means the pool was saturated almost the entire run — the ramp was instant and there was no tail. If average sits at 1.2 while peak is 5, your workers overlap briefly but spend most of the run idle or queued. Both numbers matter; peak tells you the pool can overlap, average tells you it usually does.

To find which requests produced the peak, keep the paths alongside the running count:

active = set()
for ts, delta, path in events:
    if delta == 1:
        active.add(path)
    else:
        active.discard(path)
    if len(active) == peak:
        print("peak overlap:", sorted(active))

For the curl burst that prints peak overlap: ['/products/1', '/products/2', '/products/3', '/products/4', '/products/5'] — all five, which is what you expect from five parallel curls. When the same analysis runs against a broken scraper, you’ll see the peak pair and, more usefully, the long stretches where active contains exactly one path.

Bucket the Timeline into Per-Second In-Flight Counts

A single max and average hide the shape of the run. Bucket the timeline into one-second slices and you can see the ramp-up, whether the pool holds saturation, and where it collapses — usually at the tail, when the URL queue drains and workers starve.

import math

def per_second(samples, peak):
    if not samples:
        return
    t0 = math.floor(samples[0][0])
    t_end = math.floor(samples[-1][0]) + 1
    buckets = {}
    for sec in range(t0, t_end):
        # max count observed during this second
        counts = [c for ts, c in samples if sec <= ts < sec + 1]
        buckets[sec] = max(counts) if counts else 0
    for sec in sorted(buckets):
        print(f"second {sec - t0}: peak in-flight {buckets[sec]}")

For a healthy 5-worker crawl of 25 product pages against the 800ms listener, the buckets look like:

SecondPeak in-flight
01
13
25
35
45
52

One second of ramp, three seconds of full saturation, a short tail. That’s the fingerprint of a pool that’s actually working. Compare it to the same crawl from the locked-session scraper:

SecondPeak in-flight
01
11
21
…1
191

Flat line at one. Twenty workers configured, and the timeline is indistinguishable from workers=1.

I’d argue the per-second view is more useful than the peak number alone, and this is where I’d push back on the common practice of reporting only “max concurrency” in load-test output. A pool that spikes to 20 for 200ms and then settles at 3 will report a max of 20 and still be mostly serial in practice. The bucketed timeline is honest in a way a single scalar isn’t. If you only look at one output from this whole exercise, look at the buckets.

Diagnose the Classic Failure: In-Flight Stuck at One

Run the measurement against the locked-session crawler from the first section and you get this:

requests:        50
max in-flight:   1
avg in-flight:   1.00
wall-clock span: 40.05s

Before the fix — one start, one end, alternating for the entire run:

{"event": "start", "ts": 1730000000.112, "path": "/products/1"}
{"event": "end",   "ts": 1730000000.913, "path": "/products/1"}
{"event": "start", "ts": 1730000000.914, "path": "/products/2"}
{"event": "end",   "ts": 1730000001.715, "path": "/products/2"}
{"event": "start", "ts": 1730000001.716, "path": "/products/3"}

The fix depends on why it serialized. For the shared-lock pattern, the answer is to stop sharing one session across threads — or stop using threads:

 import threading
 import requests
 
-session = requests.Session()
-session_lock = threading.Lock()
-
 def fetch(url):
-    with session_lock:
-        resp = session.get(url, timeout=10)
-    return resp.text
+    # one Session per worker, created in worker init, or:
+    resp = requests.get(url, timeout=10)
+    return resp.text

Or drop threads entirely and go async, where the concurrency is structural rather than bolted on:

import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def main(urls):
    connector = aiohttp.TCPConnector(limit=20)   # explicit pool cap
    async with aiohttp.ClientSession(connector=connector) as session:
        results = await asyncio.gather(*(fetch(session, u) for u in urls))
    return results

After the fix, the same crawl produces:

requests:        50
max in-flight:   5
avg in-flight:   4.91
wall-clock span: 8.21s
{"event": "start", "ts": 1730000000.112, "path": "/products/1"}
{"event": "start", "ts": 1730000000.113, "path": "/products/2"}
{"event": "start", "ts": 1730000000.115, "path": "/products/3"}
{"event": "start", "ts": 1730000000.116, "path": "/products/4"}
{"event": "start", "ts": 1730000000.118, "path": "/products/5"}
{"event": "end",   "ts": 1730000000.914, "path": "/products/1"}
{"event": "end",   "ts": 1730000000.915, "path": "/products/2"}

Five starts in 6ms. Wall time drops from 40s to 8.2s, which is close to the theoretical floor for 50 requests at 800ms each across 5 workers. The measurement confirms the fix rather than trusting it.

The same harness catches the subtler variants. A connection pool capped at 2 shows peak in-flight of 2 with workers waiting on sockets. A retry backoff that holds a global lock during sleep shows concurrency collapse after the listener returns a few 429s — add a --reject flag to the handler if you want to reproduce that. Each failure mode has a distinct bucket shape, and after you’ve seen two or three of them, you’ll start reading the timeline like a chart of the scraper’s pulse.

Know the Limits: What a Local Listener Cannot Prove

Be honest about what this measures. The listener proves your client can issue overlapping requests. It proves nothing about what happens against a real origin.

Locally measuredOnly visible against the live target
Client-side overlap of requestsRTT and bandwidth effects on timing
Worker pool saturationServer-side throttling and 429 behavior
Locks, session serialization, blocking callsKeep-alive and connection reuse with the real server
Retry logic firing pattern (with a --reject flag)Geographic routing and CDN edge behavior
Queue starvation at the tail of a crawlAnti-bot responses masquerading as 200 OK

Two of those right-hand items deserve emphasis. First, connection reuse: against the listener, every curl is a fresh connection, while against a real origin your session may keep one connection alive and pipeline over it — the in-flight picture can look different in ways that matter for throughput. Second, block pages. A scraper that “successfully” fetches 50 pages concurrently against the live site may be receiving 50 block pages in parallel, which the local listener can’t warn you about — that’s a separate failure mode with its own symptoms, covered in how 200 OK responses can actually be block pages.

To at least approximate real-world latency, add a delay flag to the listener and stress the pool at different speeds:

parser.add_argument("--delay", type=float, default=0.8,
                    help="simulated origin response time in seconds")
# in the handler:
time.sleep(server.delay)

Run your crawl with --delay 0.2, then --delay 2.0. At 2 seconds, a 5-worker pool has to sustain 5 in-flight just to keep up; if average in-flight sags below the worker count at high delay, your workers are spending real time somewhere other than the network — parsing, disk writes, GIL contention — and that’s worth knowing before you blame the target site for slow crawls. If you want to go deeper on separating network time from processing time, the scrape latency timing harness breaks the request lifecycle into measurable phases.

One more limit worth stating plainly: this technique measures your scraper, not your infrastructure. If you’re running through a scraping API or a proxy fleet, the concurrency you observe locally is your submission concurrency. What the provider does with those submissions — queueing, per-domain throttling, retry backoff — is a black box from your side, and you should verify it separately, for example by checking how many async jobs sit in processing at once via the jobs endpoint rather than assuming submission rate equals execution rate. For more on that distinction, see client-side vs service-side retries.

Wrap-Up

Configured concurrency is a claim. Measured concurrency is a fact. The listener-and-sweep harness turns the claim into a fact in about fifty lines of code, and it catches the serialization bugs — shared locks, blocked event loops, capped connection pools — that no amount of log inspection will surface from inside the scraper.

The workflow worth committing to memory:

  1. Run the listener with a delay that mimics your target’s response time.
  2. Redirect your crawler’s base URL via environment variable.
  3. Sweep the JSONL log for max and average in-flight.
  4. Bucket per second and check the pool actually saturates.
  5. Only then point the scraper at the real target — and separately verify that the responses you’re getting back are real content, not polite refusals.

Do this once per crawler, and again any time you touch the fetch layer. It’s the cheapest load test you will ever run, and it’s the only one that answers the question that actually matters: how many requests are on the wire, right now, at the same time.

#web scraping #concurrency #python #load testing #worker pools #slot:measurement

Related Articles