Too Many Open Files: Fix a Socket Leak in Your Scraper

A Leak That Passes Every Smoke Test

Somewhere around page 1,012, the worker stopped talking. The log showed a steady stream of fetched product pages, then one last line — fetching https://store.example.com/products/1042 — followed by a traceback ending in OSError: [Errno 24] Too many open files and a dead process. No timeout, no 403, no rate limit. The kernel simply refused to hand the process another socket.

The infuriating part: the same script passed every smoke test. Fifty pages here, a hundred there, all green. The bug hides at small counts because it’s arithmetic, not logic — the worker leaked roughly one file descriptor per request against a default soft limit of 1,024, so anything under a thousand fetches looks flawless. What follows is the full path: confirm the leak with lsof, find the two lines of code causing it, fix them with a pooled session and closed responses, and put a guard in place so it cannot quietly return.

Symptoms: EMFILE Errors That Only Surface Mid-Batch

Here is the failure signature. A worker that had been running clean for twenty minutes suddenly dies inside the fetch call, deep in urllib3’s socket creation:

Traceback (most recent call last):
  File "worker.py", line 48, in fetch_page
    r = requests.get(url, timeout=10)
  File ".../requests/api.py", line 73, in get
    return request("get", url, **kwargs)
  File ".../requests/sessions.py", line 602, in request
    resp = self.send(prep, **send_kwargs)
  ...
  File ".../urllib3/util/connection.py", line 93, in create_connection
    sock = socket.socket(af, socktype, proto)
OSError: [Errno 24] Too many open files

Errno 24 is EMFILE: the process hit its own cap on open file descriptors. Every socket, open file, pipe, and log handle counts against the same limit. (Its cousin ENFILE, errno 23, is the system-wide table — if you see that, the problem is bigger than one process.) Sockets are file descriptors, which is why an HTTP-heavy workload trips a limit named after files.

Now put the crash next to the process limit:

$ ulimit -n
1024
$ grep -c "fetched ok" worker-3.log
1011
$ tail -n 2 worker-3.log
worker-3 fetching https://store.example.com/products/1042
worker-3 crashed: OSError(24, 'Too many open files')

That is not a coincidence, it is a receipt. The worker started with 12 open descriptors (interpreter, stdio, log file). One leaked socket per request means descriptor 1,024 gets requested on fetch number 1,012. The math lines up almost exactly with the crash.

This is why the leak survived testing. A 50-page smoke test peaks around 62 descriptors and never comes close to the ceiling. The failure only exists at the scale your batch jobs actually run at, which is precisely where you don’t want to discover it.

Confirm the Leak: Counting Open Sockets with lsof

Before touching code, prove the leak exists and understand its shape. Grab the worker’s PID and look at what it holds:

lsof -nP -p 24180 | grep TCP | head -5

python3 24180 scraper 15u IPv4 1284931 0t0 TCP worker-1:45122->store.example.com:https (CLOSE_WAIT)
python3 24180 scraper 16u IPv4 1284932 0t0 TCP worker-1:45124->store.example.com:https (CLOSE_WAIT)
python3 24180 scraper 17u IPv4 1284933 0t0 TCP worker-1:45126->store.example.com:https (CLOSE_WAIT)
...596 more lines like this

The state matters more than the count. CLOSE_WAIT means the remote end sent a FIN — the server closed its side — and your process acknowledged it but never called close() on its own end. The kernel keeps that descriptor alive until the process exits or closes it. CLOSE_WAIT sockets do not expire on their own; they are permanent tenants.

Distinguish this from TIME_WAIT, which you’ll also see under heavy traffic. TIME_WAIT means your side closed first and the kernel is holding the port briefly as a safety measure — those clear themselves in about a minute. Thousands of TIME_WAIT sockets indicate churn, not a leak. Hundreds of CLOSE_WAIT sockets indicate an application that forgot to close what it opened.

Next, sample the descriptor count while the worker runs and correlate it with pages fetched:

while kill -0 "$PID" 2>/dev/null; do
  printf '%s pages=%s fds=%s\n' "$(date +%T)" "$(grep -c 'fetched ok' worker-3.log)" \
    "$(ls /proc/$PID/fd | wc -l)"
  sleep 30
done

Every 200 pages, the picture is the same:

Pages fetched	Open FDs	Net change
0	12	—
200	212	+200
400	412	+200
600	612	+200

One descriptor per request, perfectly linear, no plateau. A healthy pooled client plateaus; a leak climbs forever. You now have proof, a rate, and a signature (CLOSE_WAIT) pointing at unclosed sockets. Time to read the code.

Root Cause: A New Connection Per Request and Responses Never Closed

The worker loop, stripped to the relevant parts:

import requests
from bs4 import BeautifulSoup

def fetch_page(page_id: int) -> dict:
    url = f"https://store.example.com/products/{page_id}"

    # LEAK 1: requests.get builds a throwaway Session per call,
    # which opens a fresh TCP + TLS connection every single time.
    r = requests.get(url, timeout=10)

    if r.status_code != 200:
        # LEAK 2: error-path body is never read and the response is
        # never closed — the socket below it stays pinned open.
        return {"id": page_id, "error": r.status_code}

    soup = BeautifulSoup(r.text, "html.parser")
    return {"id": page_id, "title": soup.h1.get_text(strip=True) if soup.h1 else None}

Two defects compound here.

Defect 1: a new connection per request. requests.get() is a convenience wrapper that constructs a temporary Session, sends one request, and abandons the session. No connection is ever reused, so every page costs a full TCP handshake and TLS negotiation. Cleanup of those abandoned sockets is left to the garbage collector, and GC is not a resource-management strategy — under a tight fetch loop, allocation outpaces collection, and descriptors pile up faster than finalizers release them.

Defect 2: responses never closed or consumed. On any non-200 response, the body is never read. urllib3 will not return a connection with unread data to its pool, so that socket stays held by a response object nobody references. Meanwhile the server’s keep-alive timer expires, the server sends a FIN, and the socket moves to CLOSE_WAIT — where it sits, exactly as lsof showed. A site that returns a slice of 404s and 429s across a 10,000-page run manufactures these constantly.

Together they produce the observed one-descriptor-per-request growth. Neither is exotic. Both fit on one line each, which is why code reviews miss them.

Fix 1: Pool Connections with One Session and HTTPAdapter

Build one session per worker, mount an adapter with an explicit pool, and reuse it for the process’s lifetime:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from bs4 import BeautifulSoup

def build_session(pool_size: int = 10) -> requests.Session:
    retry = Retry(
        total=3,
        backoff_factor=0.5,
        status_forcelist=(429, 500, 502, 503),
        allowed_methods=frozenset(["GET"]),
    )
    adapter = HTTPAdapter(
        pool_connections=pool_size,   # number of host pools to cache
        pool_maxsize=pool_size,       # sockets kept alive per pool
        max_retries=retry,
    )
    session = requests.Session()
    session.mount("https://", adapter)  # or scope tighter: "https://store.example.com"
    return session

SESSION = build_session(pool_size=10)

def fetch_page(page_id: int) -> dict:
    url = f"https://store.example.com/products/{page_id}"
    with SESSION.get(url, timeout=10) as r:   # context manager — see Fix 2
        r.raise_for_status()
        soup = BeautifulSoup(r.text, "html.parser")
        return {"id": page_id, "title": soup.h1.get_text(strip=True) if soup.h1 else None}

The pool sizing rule is simple: pool_maxsize should match your concurrent worker count. Ten workers, pool of ten. If the pool is smaller than your concurrency, urllib3 logs “connection pool is full, discarding connection” and closes the overflow — that’s not a leak, but you lose the reuse you built the pool for. If you run multiprocessing rather than threads, each worker process needs its own session; pools don’t cross process boundaries.

The Retry configuration deserves a note. Where you do the retrying matters — client-side retries with backoff behave very differently from retries the service performs for you, and mixing both blindly can multiply your effective request count. I wrote about that distinction in Client-Side vs Service-Side Retries for Scraping Failures.

Here’s what changed across a 1,000-page run against the same storefront, measured from one host (your absolute numbers will differ; the ratio won’t):

Metric, 1,000 fetches	requests.get per call	Pooled Session
TCP connections opened	1,000	10
p95 latency per page	412 ms	186 ms
FDs held after run	1,012	14

The latency win is not a bonus, it’s the same fix. Every discarded connection was also a discarded TLS handshake. Pooling removes both the descriptor leak and the per-page handshake tax in one change, which is why I consider a session-per-process non-negotiable in any scraper that runs more than a handful of requests.

Fix 2: Close Every Response with Context Managers

Pooling fixes connection creation. The second half is guaranteeing every response gets released, including the paths where errors interrupt a fetch. The with statement does this deterministically:

def fetch_page_streamed(page_id: int) -> bytes:
    url = f"https://store.example.com/products/{page_id}"
    with SESSION.get(url, stream=True, timeout=10) as r:
        r.raise_for_status()
        buf = bytearray()
        for chunk in r.iter_content(chunk_size=64 * 1024):
            buf.extend(chunk)
            if len(buf) > 4 * 1024 * 1024:   # abort oversized pages early
                raise ValueError(f"page {page_id} exceeds 4 MiB")
    return bytes(buf)

If the size cap raises mid-download, the context manager still closes the response on the way out. That’s the whole point — the cleanup runs whether the block succeeds, raises, or the loop bails early.

Some code paths can’t use a context manager, say a prefetch function returning the response object to a caller. Use an explicit finally:

r = SESSION.get(url, stream=True, timeout=10)
try:
    r.raise_for_status()
    first = next(r.iter_content(chunk_size=64 * 1024), b"")
    return sniff_content_type(first)
finally:
    r.close()

Understand why stream=True without a close is the worst offender in this entire class of bugs. Streaming defers body consumption — that’s its purpose. If you read part of the body, or none of it, and never close, the socket sits there holding unread bytes. The pool won’t check it out, the server eventually gives up and FINs it, and you’ve manufactured another permanent CLOSE_WAIT resident. A normal non-streaming response whose body gets fully read via .text or .content is released back to the pool automatically; a half-read streaming response never is. Streaming buys you memory efficiency on large pages, and charges you the obligation to close.

Verify the Fix: FD Counts Before and After a 10,000-Page Run

Use the same measurement that found the leak to prove it’s gone. The script below samples the process’s own descriptor directory every 500 pages:

import os

def fd_count() -> int:
    return len(os.listdir("/proc/self/fd"))   # Linux; on macOS use lsof -p $$

def run_batch(page_ids, fetch):
    base = fd_count()
    print(f"baseline fds={base}")
    for n, page_id in enumerate(page_ids, 1):
        fetch(page_id)
        if n % 500 == 0:
            print(f"pages={n:6d} fds={fd_count():4d}")

One wrinkle: to measure the leaking code all the way to 10,000 pages, raise the limit first (ulimit -n 65535 in the shell that launches it) — otherwise the old code dies near page 1,012 again, which is diagnostic but short-lived. The results, leaking code versus fixed code:

Pages	Leaking worker	Fixed worker
0	12	12
5,000	5,012	13
10,000	10,012	14

Fixed code drifts by two descriptors across ten thousand fetches — log rotation, mostly — and plateaus. Leaking code climbs at exactly one per page with no ceiling behavior at all.

Two practical notes. If you’d rather not hammer a live site for verification, point the harness at a local fixture server serving saved HTML; descriptor behavior is identical. And if you want the run to also tell you about fetch quality, not just descriptor hygiene, the same harness doubles as a success-rate measurement — see Measure Your Scraper’s Real Success Rate Locally for wiring that up.

Guardrails: Raise File Limits and Add a CI Leak Assertion

Two defenses keep this from recurring, and only one of them is about limits.

First, tune the runtime limit for legitimate concurrency. A scraper doing 10 concurrent workers with a pooled session needs perhaps 30 descriptors total — but your worker also opens files, logs, and pipes, and 1,024 is a cramped default for any long-running service. If the worker runs under systemd:

# /etc/systemd/system/scraper.service.d/limits.conf
[Service]
LimitNOFILE=65535

sudo systemctl daemon-reload
sudo systemctl restart scraper

Be clear about what this is: headroom, not a fix. A linear leak defeats any number you pick — raising the limit just relocates the cliff from page 1,012 to page 65,524 and lets three more hours of batch work corrupt before the crash. I have seen teams raise LimitNOFILE three times on the same service before anyone ran lsof. Don’t be that team. Raise it once, for the right reason, and assert the leak is gone.

Which is the second defense — a regression test in CI:

import os
import pytest

from scraper import build_session   # the fixed module under test

@pytest.mark.skipif(not os.path.isdir("/proc/self/fd"), reason="procfs required")
def test_hundred_fetches_do_not_leak_fds():
    session = build_session(pool_size=2)
    session.get("https://example.com", timeout=10)   # warm imports + one connection

    baseline = len(os.listdir("/proc/self/fd"))
    for _ in range(100):
        with session.get("https://example.com", timeout=10) as r:
            r.raise_for_status()
            r.content

    assert len(os.listdir("/proc/self/fd")) == baseline

The warm-up fetch matters: lazy imports and warning machinery open files on first use, and you want the baseline taken after all that settles. If exact equality flakes in your environment, allow one or two descriptors of slack — a real leak on this test shows up as +100, not +2. If CI reaching the network bothers you (it should), serve a local fixture directory and point the test at it; the descriptor mechanics don’t care whether the peer is across the ocean or on localhost.

When your process shouldn’t hold the sockets at all

There’s a structural escape hatch worth naming: if the fetching itself moves off your box, your worker holds zero scrape sockets. FineData’s async batch endpoint takes the entire URL list, fans it out on its own workers, and calls your webhook when the batch completes:

import requests

API = "https://api.finedata.ai"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}

def submit_batch(page_ids):
    payload = {
        "callback_url": "https://ops.example.com/hooks/batch-done",
        "requests": [
            {"url": f"https://store.example.com/products/{pid}", "formats": ["markdown"]}
            for pid in page_ids
        ],
    }
    resp = requests.post(f"{API}/api/v1/async/batch", json=payload,
                         headers=HEADERS, timeout=30)
    resp.raise_for_status()
    return resp.json()   # record the batch id for tracking

def check_progress():
    resp = requests.get(f"{API}/api/v1/async/jobs",
                        params={"status": "processing", "limit": 50},
                        headers=HEADERS, timeout=30)
    return resp.json()

My take: this is the right call when volume is high or the target is hostile and you’d otherwise be maintaining proxy rotation, browser fingerprinting, and retry policy yourself — see Sync vs Async Scraping: When Blocking Requests Are Enough for the decision framework. For a modest internal job against a friendly site, a pooled session is cheaper and simpler, and I’d start there. Offloading also swaps your socket problem for a webhook-handling problem, which is a fair trade at scale and an annoying one at small scale.

Wrap-Up

The whole failure decomposes into three checkable facts: errno 24 at a predictable page count, CLOSE_WAIT sockets visible in lsof, and descriptor growth of exactly one per request. The fix decomposes just as cleanly — one pooled Session sized to your worker count, and every response closed via context manager, especially the streaming and error paths. Verify with /proc/self/fd before and after a long run, then pin it with a CI assertion so the next refactor can’t reintroduce it quietly.

If your worker is crashing mid-batch right now, run the lsof check before you change anything. Ten seconds of measurement will tell you whether you have this exact bug, how fast it leaks, and whether your fix actually worked — and if you’d rather not own ten thousand sockets in the first place, FineData can hold them while your worker stays thin.