Skip to content
Technical 16 min read

Measure How Often Your Scraper Reopens a Connection

Run a local listener that counts TCP handshakes to reveal how often your scraper reopens connections — and what each reconnect adds in time and CPU.

FE
FineData Engineering · Editorial Policy
|
On this page

Why Each Extra TCP Handshake Costs Time and CPU Before Any Data Moves

A scraper that opens a fresh connection for every request is doing three pieces of unpaid work before a single byte of page content moves: a DNS lookup, a TCP three-way handshake, and (on HTTPS) a TLS handshake with asymmetric crypto. None of that work fetches data. It is pure setup cost, and on a crawl of a hundred thousand pages it compounds into real wall time, real CPU, and real socket churn — the same churn that eventually bites you as Too many open files if you leak the sockets on top of reopening them.

Here is what the setup work typically looks like against a real origin, compared to a reused keep-alive connection:

StepNew connection to store.example.comReused keep-alive connection
DNS lookup20–120 ms (cached: ~0–5 ms)0 ms
TCP three-way handshake1× RTT (~20–80 ms cross-continent)0 ms
TLS 1.3 handshake1× RTT + key exchange (~25–100 ms)0 ms
First byte / request itselfSameSame

You can measure the reconnect overhead directly. Make one request on a fresh connection, then a second on the same connection, and subtract:

import http.client
import time

def timed_get(conn, path):
    start = time.perf_counter()
    conn.request("GET", path)  # http.client sets the Host header for you
    resp = conn.getresponse()
    resp.read()
    return (time.perf_counter() - start) * 1000

conn = http.client.HTTPSConnection("store.example.com", timeout=30)

first = timed_get(conn, "/product/1")
second = timed_get(conn, "/product/2")
conn.close()

print(f"first request (new connection): {first:.1f} ms")
print(f"second request (reused):        {second:.1f} ms")
print(f"per-reconnect overhead:         {first - second:.1f} ms")

The delta is what every avoidable reconnect costs you. Run it a few times and take the median — the first request also eats a cold DNS cache, so a single run overstates things.

The problem is that most scrapers don’t know they’re reconnecting. The client library hides it. Logs show successful 200s at reasonable latency. The only way to see the pattern is to watch the TCP layer, and the cheapest way to do that is a local listener that counts handshakes while your real scraper code runs unchanged against it.

Build a Local Listener That Counts Every TCP Handshake

The listener below does one job: every time accept() fires, increment a counter and print a timestamped line. It also speaks just enough HTTP — with real keep-alive — so that real client libraries behave the way they would against a real origin.

import asyncio
import time

HOST, PORT = "127.0.0.1", 8080
HANDSHAKES = 0
REQUESTS = 0

BODY = b"""<html><body>
<ul>
<li><a href="/product/1">Widget A</a></li>
<li><a href="/product/2">Widget B</a></li>
<li><a href="/product/3">Widget C</a></li>
</ul>
</body></html>"""

def build_response(status_line, body):
    return (
        status_line
        + b"Content-Type: text/html\r\n"
        + b"Connection: keep-alive\r\n"
        + f"Content-Length: {len(body)}\r\n".encode()
        + b"\r\n"
        + body
    )

RESPONSE_200 = build_response(b"HTTP/1.1 200 OK\r\n", BODY)
RESPONSE_500 = build_response(b"HTTP/1.1 500 Internal Server Error\r\n", b"retry")

FLAKY_PATH = "/product/7"  # returns 500 once per connection, for retry testing

async def read_request(reader):
    """Read one request head. Returns (path, connection) or None on EOF.

    Uses strip() so both CRLF and bare-LF line endings terminate a header,
    and treats an empty read (b"") as EOF rather than end-of-headers.
    """
    path = None
    connection = "keep-alive"
    while True:
        line = await reader.readline()
        if line == b"":
            return None  # client closed the connection
        stripped = line.strip()
        if stripped == b"":
            if path is None:
                continue  # stray blank line before the request line
            return path, connection
        if path is None:
            path = stripped.split(b" ")[1].decode()  # "GET /path HTTP/1.1"
        else:
            name, _, value = stripped.partition(b":")
            if name.strip().lower() == b"connection":
                connection = value.strip().decode().lower()

async def handle(reader, writer):
    global HANDSHAKES, REQUESTS
    HANDSHAKES += 1
    conn_id = HANDSHAKES
    peer = writer.get_extra_info("peername")
    print(f"[{time.strftime('%H:%M:%S')}] handshake #{conn_id} from {peer}")
    flaky_served = False
    try:
        # Outer loop: serve as many requests as the client sends on this
        # one connection. We only exit on EOF, a "Connection: close"
        # request, or an error — never after a single response.
        while True:
            request = await read_request(reader)
            if request is None:
                break  # client hung up; that is the client's choice, not ours
            path, connection = request
            REQUESTS += 1
            if path == FLAKY_PATH and not flaky_served:
                flaky_served = True
                writer.write(RESPONSE_500)
            else:
                writer.write(RESPONSE_200)
            await writer.drain()
            print(f"[{time.strftime('%H:%M:%S')}] "
                  f"request #{REQUESTS} ({path}) on connection #{conn_id}")
            if connection == "close":
                break
    except (ConnectionResetError, BrokenPipeError):
        pass
    finally:
        writer.close()

async def main():
    server = await asyncio.start_server(handle, HOST, PORT)
    print(f"listening on {HOST}:{PORT}")
    async with server:
        await server.serve_forever()

asyncio.run(main())

The outer per-request loop is what makes the instrument honest. If the listener closed the socket after each response, every client would be forced to reconnect and the measurement would be meaningless. Here the connection stays open until the client closes it (or explicitly asks us to with Connection: close), which is what lets you observe whether the client chooses to reuse it. Note also that read_request distinguishes a true EOF (b"") from the blank line that ends a header block, and tolerates both \r\n and bare \n line endings — this is an instrument people will copy verbatim, so it shouldn’t mis-parse a slightly non-conforming client.

Probe it with three separate one-shot connections (e.g. three curl http://127.0.0.1:8080/ invocations, or a script that opens and closes three sockets) to confirm the counter works:

listening on 127.0.0.1:8080
[14:02:11] handshake #1 from ('127.0.0.1', 51742)
[14:02:11] request #1 (/) on connection #1
[14:02:14] handshake #2 from ('127.0.0.1', 51768)
[14:02:14] request #2 (/) on connection #2
[14:02:16] handshake #3 from ('127.0.0.1', 51790)
[14:02:16] request #3 (/) on connection #3

Three connections, three handshake lines, three distinct ephemeral ports — and one request each, because each one-shot client hung up after its response. That’s the instrument working. Now point real scraper code at it.

Point Your Scraper at the Listener with a Hosts Entry and a Dummy HTTP Responder

You don’t want to maintain a special “test mode” branch inside your scraper — that code path drifts from production and you end up measuring a strawman. Instead, map the hostname your scraper already targets onto the loopback interface and let the real code run untouched:

127.0.0.1    store.example.com

That entry goes in /etc/hosts (or C:\Windows\System32\drivers\etc\hosts on Windows). Now store.example.com resolves to 127.0.0.1, and anything your scraper sends to port 80 lands on your machine.

If you can’t run the listener on port 80 directly (binding low ports needs root), redirect instead. On Linux:

sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 80 \
  -j REDIRECT --to-port 8080

On macOS, a pfctl rdr rule does the same job. Don’t insert an intermediate proxy like a forking socat unless you’ve verified it forwards connections one-to-one — anything that pools upstream connections will corrupt your handshake counts.

One honest caveat: against a local listener over plain HTTP you lose visibility into the TLS handshake cost, because there isn’t one. That’s fine. What you’re measuring is the reconnect pattern — how often the client opens a socket at all — and that pattern is identical with or without TLS. The per-reconnect millisecond cost from the first section is what you multiply the count by.

The dummy responder (embedded in the listener above) is deliberately shaped so your parsers run unmodified: a real status line, Connection: keep-alive, a correct Content-Length, and a body with fake product links in the markup your extraction rules expect. If your scraper’s link extractor finds /product/1 through /product/3, the whole pipeline downstream of the socket behaves normally. The one baked-in quirk — a single 500 for /product/7 per connection — exists so you can verify that retry logic reuses the connection instead of tearing it down.

Run a Baseline Crawl and Record the Handshake-to-Request Ratio

Here is the anti-pattern that produces a 1:1 reconnect ratio. It’s extremely common in quick scripts, cron jobs, and — I see this constantly — inside data-pipeline tasks that were “temporary” two years ago:

import requests

def crawl_baseline(product_ids):
    results = []
    for pid in product_ids:
        resp = requests.get(
            f"http://store.example.com/product/{pid}",
            timeout=30,
        )
        results.append((pid, resp.status_code, len(resp.text)))
    return results

results = crawl_baseline(range(1, 21))  # 20 requests
print(f"requests: {len(results)}")

Every requests.get() call constructs a fresh session, opens a new socket, fetches one page, and throws the connection away. The listener sees all of it:

listening on 127.0.0.1:8080
[14:10:01] handshake #1  from ('127.0.0.1', 51822)
[14:10:01] request #1 (/product/1) on connection #1
[14:10:01] handshake #2  from ('127.0.0.1', 51834)
[14:10:01] request #2 (/product/2) on connection #2
[14:10:01] handshake #3  from ('127.0.0.1', 51840)
[14:10:01] request #3 (/product/3) on connection #3
[14:10:01] handshake #4  from ('127.0.0.1', 51846)
... (connections 5 through 19 omitted)
[14:10:02] handshake #20 from ('127.0.0.1', 51998)
[14:10:02] request #20 (/product/20) on connection #20

Twenty request lines, twenty handshake lines — every request on its own connection, and every connection torn down by the client immediately after. A 1:1 reconnect ratio. Against localhost each handshake is nearly free, which is exactly why nobody notices in testing — and exactly why the production version against a real origin feels slow and burns CPU for no reason. The ratio is the number that matters, not the local latency.

Record this baseline. Handshake count, wall time, CPU time. Everything after this is judged against it.

Diagnose the Reconnect Pattern: Per-Request, Pool Exhaustion, or Idle Keep-Alive Expiry

A raw handshake count tells you that you’re reconnecting. The timestamps tell you why. Read the gaps between log lines before you change any code.

Here’s an annotated excerpt from a crawler that runs 10 concurrent workers over 100 requests:

[14:22:01] handshake #1  from ('127.0.0.1', 52244)   # crawl starts, 10 workers connect
[14:22:01] handshake #2  ...
[14:22:01] handshake #10 ...                          # 10 handshakes, then requests
                                                      # keep flowing on those connections
[14:22:44] handshake #11 ...                          # worker picks up queued URL
                                                      # after ~90s idle
[14:22:44] handshake #12 ...
[14:22:44] handshake #20 ...                          # all 10 workers reconnect

Ten handshakes up front, then a burst of ten more after a roughly 90-second quiet period. That’s not per-request reconnecting — that’s keep-alive expiry. The pooled connections sat idle longer than the client’s (or an intermediary’s) keep-alive limit, got closed, and every worker had to re-establish on the next wave of URLs. The fix is entirely different from the per-request case: you either raise keepalive_expiry or you stop letting workers idle out mid-run.

Use the table to classify what your own log shows:

Observed symptomRoot causeMatching fix
1 handshake per request (ratio 1:1)No connection reuse at all — bare requests.get() per URLShared Session or Client (next section)
Handshakes equal to concurrency level, then quietCorrect pooling — this is healthy, not a bugNone
Handshake bursts clustered after idle gapsKeep-alive expiry during lulls in the queueRaise keepalive_expiry, or keep the queue from draining mid-run
Handshakes spike with error responsesServer closing connections on 4xx/5xx, or client tearing down pool on exceptionIsolate error handling so it doesn’t discard the pool
Handshakes grow over a long runPool churn from per-domain limits or a slow socket leakCap pool size, check fd counts (fix the socket leak)

Note the second row, because it’s the one people misread. If you run 10 workers and see exactly 10 handshakes total, your pooling is working. Chasing a lower count than your concurrency level is a waste of time — over HTTP/1.1, you can’t share one TCP connection across concurrent requests, because the responses would interleave. (HTTP/2 multiplexing does change this: with httpx you can pass http2=True — provided you’ve installed the h2 extra — and one connection can carry many concurrent streams. But most scraping targets and most scraper stacks are still HTTP/1.1, and the concurrency floor below stands for it.)

Switch to a Persistent Session and Cap the Connection Pool

The fix for the 1:1 baseline is mechanical. Before:

# per-call: new connection every request
for pid in product_ids:
    resp = requests.get(f"http://store.example.com/product/{pid}", timeout=30)

After, with requests:

# shared session: connections reused across calls
import requests

def crawl_with_session(product_ids):
    with requests.Session() as session:
        session.headers["User-Agent"] = "my-scraper/1.0"
        results = []
        for pid in product_ids:
            resp = session.get(
                f"http://store.example.com/product/{pid}",
                timeout=30,
            )
            results.append((pid, resp.status_code, len(resp.text)))
        return results

Or, if you’re on httpx, with explicit pool limits so reuse doesn’t degrade into unbounded connection sprawl under concurrency:

import httpx

limits = httpx.Limits(max_keepalive_connections=10, keepalive_expiry=120)

def crawl_with_client(product_ids):
    with httpx.Client(limits=limits, timeout=30) as client:
        results = []
        for pid in product_ids:
            resp = client.get(f"http://store.example.com/product/{pid}")
            results.append((pid, resp.status_code, len(resp.text)))
        return results

I prefer the httpx version for anything long-running, purely because keepalive_expiry is a first-class knob. requests.Session reuses connections but gives you little control over when an idle one is dropped, which matters once you’re diagnosing the expiry pattern from the previous section.

For Scrapy crawls, the same tuning lives in settings. The download slot per domain is where connection reuse is actually governed:

# settings.py
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.25
# Scrapy keeps persistent connections per slot. One measurement caveat:
# HTTPCACHE_ENABLED defaults to False, but if you've turned it on for
# development, turn it back off for these runs or you'll measure cache
# hits instead of connection behavior.

One opinion that some people push back on: cap the pool below your target’s comfort level even if you could go higher. A pool of 10 keep-alive connections with a small delay between requests draws far less attention than 50 simultaneous connections hammering one origin, and for most crawl workloads the throughput difference is negligible. You’re trading a few percent of speed for a meaningfully lower chance of getting rate-limited or blocked. That trade is almost always worth it.

Re-Run the Listener and Compare Handshakes, Wall Time, and CPU

Same listener, same 20 URLs, same machine — only the client code changed. Wrap each crawl with a harness that captures all three metrics:

import time

def measure(crawl_fn, *args):
    wall_start = time.perf_counter()
    cpu_start = time.process_time()
    results = crawl_fn(*args)
    wall = time.perf_counter() - wall_start
    cpu = time.process_time() - cpu_start
    return {
        "requests": len(results),
        "wall_s": wall,
        "cpu_s": cpu,
        "ms_per_request": (wall * 1000) / len(results),
    }

Then run both crawls through it. Against the local listener the wall time barely moves — localhost handshakes are microseconds — so don’t expect a dramatic local speedup. The measurement that matters locally is the handshake count, and the CPU delta shows up as the connection-setup work (especially the crypto once you extrapolate to TLS) that disappears:

Metric (20 sequential requests)Before: bare requests.get()After: shared Session
Handshakes counted by listener201
Total wall time (local, plain HTTP)~1.9 s~1.8 s
Process CPU time~0.11 s~0.08 s
Average ms per request~95 ms~90 ms

The local numbers are intentionally boring. That’s the point of a local harness: it isolates the pattern, not the production penalty. Do the extrapolation explicitly. If your earlier probe showed, say, 60 ms of per-reconnect overhead against the real origin, then 19 avoidable handshakes per 20 requests means roughly 1.1 seconds of pure setup cost per 20 pages — about 57 ms of dead time per page, on top of a CPU cost you pay per connection for key exchange that you simply stop paying. At crawl scale, multiply it out against your daily page volume and see whether it’s worth an afternoon. Usually it is.

Watch for Silent Reconnect Triggers: Cross-Host Redirects, Errors, and Idle Expiry

Fixing the baseline doesn’t end the story. Connections get reopened for reasons that never show up in request logs, and the listener is how you catch them.

The sneakiest one is the cross-host redirect. HTTP clients pool connections per host. When a redirect hops from one hostname to another, the client can’t reuse the origin connection even if both resolve to the same server:

[14:35:02] handshake #1  from ('127.0.0.1', 52114)   # GET example.com/product/1 -> 301
[14:35:02] handshake #2  from ('127.0.0.1', 52118)   # follow to store.example.com: new pool

Two handshakes for what every log records as one request. On sites that redirect every URL once — http-to-https hops, www-vs-bare-domain canonicalization — this silently doubles your connection count and you’ll never see it in request-level metrics. The fix is to canonicalize URLs to the final host before enqueueing them, so the redirect never happens.

Errors are the other trigger. Depending on the client, a timeout or a 5xx can tear down the whole pooled connection, forcing the next request to rebuild it. Isolate failure handling so a bad page doesn’t cost you the pool:

import httpx

limits = httpx.Limits(max_keepalive_connections=10, keepalive_expiry=120)
client = httpx.Client(limits=limits, timeout=30)

def fetch(pid, attempts=3):
    for i in range(attempts):
        try:
            resp = client.get(f"http://store.example.com/product/{pid}")
            if resp.status_code >= 500 and i < attempts - 1:
                continue  # retry, but do NOT close the client
            return resp
        except httpx.TransportError:
            # covers timeouts too: httpx.TimeoutException is a
            # subclass of TransportError, so one catch suffices
            if i == attempts - 1:
                raise
    raise RuntimeError("unreachable")

And the listener’s log when both fixes hold — this is the /product/7 flaky path earning its keep, with the retry landing on the same connection that served the 500:

[14:41:10] handshake #1  from ('127.0.0.1', 52160)
[14:41:10] request #1 (/product/1) on connection #1
[14:41:10] request #2 (/product/2) on connection #1
[14:41:11] request #7 (/product/7) on connection #1     # 500 served, retried
[14:41:11] request #8 (/product/7) on connection #1     # retry: same connection
... (requests 9 through 19 on connection #1 omitted)
[14:41:12] request #20 (/product/20) on connection #1

Twenty request lines, one handshake line. That is what a fixed crawler looks like from the TCP layer’s point of view.

Idle expiry deserves a final note. Set keepalive_expiry above your worst-case inter-request gap, not just your average one. A queue that stalls for two minutes on a slow downstream write will expire a 120-second keep-alive and reconnect the whole pool. If your pipeline has bursty phases — and most do — size the expiry for the burst gap, or add a keep-alive ping during lulls.

Wrapping Up

The method is one loop, repeated whenever something feels slow: run the listener, point the hosts entry at it, crawl a fixed set of URLs, read the handshake log, fix the pattern the timestamps reveal, re-measure. Handshake count is the metric; wall time and CPU are the receipts.

The gotchas worth remembering: keep-alive on the listener side or the measurement lies; intermediate proxies can pool connections and corrupt counts; per-host pools mean redirects cost handshakes; and concurrency-level handshake counts are healthy, not a problem. Don’t optimize below your HTTP/1.1 concurrency floor — it can’t be done without multiplexing.

If you want to go further on the measurement side, the same local-harness approach works for measuring your scraper’s real success rate and for breaking down where scrape latency actually goes. And when the target’s own defenses, not your connection handling, become the bottleneck, an async batch job that hands retries and rotation to a service is the next step up — the trade-offs between running that yourself and keeping it in-house are covered in async scraping at scale.

Reused connections won’t fix a blocked request or a broken selector. But they’re the cheapest performance win in most scrapers, they’re measurable in an afternoon, and now you have the instrument to prove it.

#connection pooling #keep-alive #tcp handshake #http clients #scraping performance #slot:measurement

Related Articles