Skip to content
Tutorial 13 min read

ENOTFOUND Under Load: Fix Your Scraper's DNS Collapse

Bursts of ENOTFOUND and EAI_AGAIN errors hit your scraper only under load. Learn to diagnose resolver overload and fix it with caching and fan-out control.

FE
FineData Engineering · Editorial Policy
|
On this page

The Symptom: DNS Errors That Appear Only Under Fan-Out

Your scraper purrs at ten requests per second for hours. Then you kick off a backfill — two hundred product pages, fired in one Promise.all — and the process sprays getaddrinfo EAI_AGAIN and ENOTFOUND errors across the log. Half a minute later the burst clears and the steady crawl works again. Nothing about the target changed: curl https://store.example.com/product/1 from the same machine, mid-burst, resolves instantly.

That signature — failures only under fan-out, clean lookups at low rate — means your DNS resolver is the component that gave up. Not the site blocking you, not your HTTP stack. The fixes people reach for first, in order: turn on heavier anti-bot flags, crank UV_THREADPOOL_SIZE, pin IPs into /etc/hosts, wrap everything in a blind retry loop. Two of those mask the symptom, one is brittle, and one actively amplifies the failure. None of them touch the bottleneck.

Reproduce the Failure: A 200-Request Burst Against store.example.com

Before fixing anything, get the failure on demand. This script fires 200 requests simultaneously and tallies failures by error code. Swap store.example.com for the host you actually scrape — the placeholder domain has no A record and will fail for a different reason than yours does.

// burst.mjs — fire 200 requests at once, tally failures by error code
const HOST = 'store.example.com';            // swap in the host you actually scrape

const codes = {};
const lat = [];
let ok = 0;

// undici wraps fetch failures in a cause chain and occasionally nests twice
// (err.cause.cause.code). Walk the whole chain; stopping at the first cause
// mislabels those burst failures as UNKNOWN.
const errorCode = (err) => {
  for (let e = err; e; e = e.cause) if (e.code) return e.code;
  return 'UNKNOWN';
};

async function hit(i) {
  const t0 = performance.now();
  try {
    const res = await fetch(`https://${HOST}/product/${i}`);
    await res.arrayBuffer();                 // drain the body
    ok++;
  } catch (err) {
    const code = errorCode(err);
    codes[code] = (codes[code] ?? 0) + 1;
  }
  lat.push(performance.now() - t0);
}

const t0 = performance.now();
await Promise.all(Array.from({ length: 200 }, (_, i) => hit(i + 1)));

const pct = (q) => lat.sort((a, b) => a - b)[Math.floor(lat.length * q) - 1];
console.log(`done in ${((performance.now() - t0) / 1000).toFixed(1)}s — ok=${ok}, ` +
            `p50=${pct(0.5).toFixed(0)}ms, p99=${pct(0.99).toFixed(0)}ms`);
for (const [code, n] of Object.entries(codes).sort((a, b) => b[1] - a[1])) {
  console.log(`  ${code.padEnd(12)} ${n}`);
}

A representative run on a laptop resolver looks like this:

$ node burst.mjs
done in 41.7s — ok=148, p50=6400ms, p99=9800ms
  EAI_AGAIN    38
  ENOTFOUND    12
  ECONNRESET   2

Now rerun it with a limiter of 5. Clean run, zero errors, faster wall clock. That contrast — broken at 200 simultaneous, healthy at 5 — is your diagnosis in one experiment. Ignore ECONNRESET for now; that’s a transport problem with its own playbook.

Also note something uncomfortable about that ENOTFOUND count: some of those are probably lies. A query the resolver drops should surface as a timeout — getaddrinfo returns EAI_AGAIN when no answer arrives — not as “name does not exist.” But resolvers under load misbehave in documented ways: Docker’s embedded DNS server has years of issue-thread reports of answering SERVFAIL, and occasionally NXDOMAIN, for queries it couldn’t actually complete, and systemd-resolved has its own backlog of load-related SERVFAIL reports. So hedge your reading: EAI_AGAIN and ENOTFOUND mixed into the same burst are one disease — resolver overload — not two.

Why Fan-Out Breaks DNS: getaddrinfo, the libuv Threadpool, and Your Router’s Resolver

Three layers fail together, and you need to understand each one before the fixes make sense.

Node’s fetch goes through undici, which opens TCP sockets via net.connect. Every socket starts with a name lookup through dns.lookup, which is not an async DNS client at all — it’s a call to the operating system’s getaddrinfo(), a blocking libc function that Node pushes onto the libuv threadpool. That pool has four threads by default, shared with fs operations and some crypto work. Two hundred simultaneous fetches queue two hundred getaddrinfo jobs on four threads while your file writes stall behind them.

MechanismWhat it actually isWhere it runsBehavior under fan-out
dns.lookup / getaddrinfoBlocking libc call, consults /etc/resolv.conflibuv threadpool (4 threads default)Jobs queue; fs competes for the same threads
dns.resolve4 / resolve6Real DNS client (c-ares), UDP/TCP to the nameserverEvent loop, no threadpoolNo queuing locally, but still hammers the upstream resolver
fetch defaultundici net.connect → dns.lookup per socketThreadpool + one lookup per socketN sockets in flight means N simultaneous getaddrinfo calls

Layer two: getaddrinfo hands each query to the system resolver — often systemd-resolved, Docker’s embedded DNS at 127.0.0.11, or your router at 192.168.1.1. These have concurrency ceilings and per-client rate limits that are generous for a laptop and miserly for a scraper. When two hundred queries land at once, the resolver drops or delays them, getaddrinfo times out, and Node surfaces EAI_AGAIN. Layer three: your retry logic fires, doubling the query volume against a resolver that was already saturated.

You can watch the threadpool effect yourself:

$ UV_THREADPOOL_SIZE=1  node burst.mjs
# lookups serialize; the resolver sees one query at a time
done in 13.2s — ok=197, EAI_AGAIN=2, ENOTFOUND=1

$ UV_THREADPOOL_SIZE=16 node burst.mjs
# 16 parallel getaddrinfo calls; the resolver chokes just as hard
done in 40.8s — ok=151, EAI_AGAIN=41, ENOTFOUND=8

Here is where I disagree with the reflex fix: bumping UV_THREADPOOL_SIZE to 16 is backwards. It relieves the local queue while pushing more simultaneous queries at a resolver that was the bottleneck all along, so the error count barely moves. If anything, UV_THREADPOOL_SIZE=1 “fixes” more, by accident, through serialization. The real answer is to stop issuing redundant lookups at all — but measure first.

Isolate DNS Latency from HTTP Latency Before Changing Any Code

Five minutes of measurement prevents a week of patching the wrong layer. Strip HTTP out entirely and time raw lookups:

// dns-probe.mjs — time raw lookups, no HTTP involved
import { promises as dns } from 'node:dns';

const samples = [];
let errors = 0;
for (let i = 0; i < 200; i++) {
  const t0 = performance.now();
  try { await dns.lookup('store.example.com'); } catch { errors++; }
  samples.push(performance.now() - t0);
  await new Promise(r => setTimeout(r, 1000));   // ~1 lookup/sec
}
const pct = (q) => samples.sort((a, b) => a - b)[Math.floor(samples.length * q) - 1];
console.log(`p50=${pct(0.5).toFixed(1)}ms p95=${pct(0.95).toFixed(1)}ms ` +
            `p99=${pct(0.99).toFixed(1)}ms errors=${errors}/200`);

Then delete the setTimeout line and fire 100 lookups inside a single Promise.all. On a strained resolver the two runs look nothing alike:

~1 lookup/sec:  p50=1.9ms p95=3.2ms p99=4.8ms errors=0/200
100 in flight:  p50=5100ms p95=8300ms p99=9900ms errors=31/100 (all EAI_AGAIN)

Cross-check with dig, which bypasses your code entirely and shows what the resolver itself reports:

# Which resolver are you actually using? Router, systemd, or container DNS?
dig +stats store.example.com | grep -i '^;; *server'

# Low rate: one query per second, 20 samples — read the "Query time" line
for i in $(seq 1 20); do
  dig +stats store.example.com | awk '/Query time/{print $4" ms"}'
  sleep 1
done

# Same query, 100 at once
seq 100 | xargs -P 100 -I{} dig +stats store.example.com 2>/dev/null \
  | awk '/Query time/{s+=$4; n++} END{printf "avg query time: %.0f ms over %d queries\n", s/n, n}'

Read it like this: if single-digit-millisecond query times at low rate turn into hundreds or thousands of milliseconds (plus timeouts) under parallel load, the resolver is confirmed as the failing component. On systemd boxes, resolvectl statistics shows cache hits and current transactions, which settles arguments fast. If you’re in Docker, suspect the embedded DNS server first — it has a low ceiling and is a notorious source of EAI_AGAIN under load. And if dig against the same resolver degrades identically, do not touch your HTTP code. It isn’t the problem.

Fix One: Cache Resolutions in the HTTP Agent Instead of Per Request

The highest-leverage change for most scrapers: stop resolving the same name two hundred times in a burst. Attach a TTL-aware DNS cache to the connection pool, so a lookup happens once per TTL window rather than once per socket:

// one-time setup, before any fetch call in the process
import { setGlobalDispatcher, Agent } from 'undici';
import CacheableLookup from 'cacheable-lookup';

const cacheable = new CacheableLookup({ maxTtl: 60_000 }); // milliseconds! 60_000 = 60s; 60 would be 60ms
let lookups = 0;
const raw = cacheable.lookup.bind(cacheable);
cacheable.lookup = (hostname, opts, cb) => { lookups++; raw(hostname, opts, cb); };
process.on('exit', () => console.error(`dns lookups: ${lookups}`));

setGlobalDispatcher(new Agent({
  connections: 64,                          // per-origin socket cap
  connect: { lookup: cacheable.lookup },    // drop-in replacement for dns.lookup
}));

Two things are working for you here. The cache serves repeated resolutions from memory, and undici’s pooled connections skip resolution entirely once a socket exists — reuse is even cheaper than a cached lookup, and worth measuring on its own (here’s a harness for that). Instrumented runs before and after the change:

$ node burst.mjs                # before: one lookup per socket
dns lookups: 200
done in 41.7s — ok=148, p50=6400ms, p99=9800ms
  EAI_AGAIN    38
  ENOTFOUND    12
  ECONNRESET   2

$ node burst.mjs                # after: one lookup per TTL window
dns lookups: 1
done in 4.1s — ok=200, p50=290ms, p99=1400ms

Two hundred lookups collapse to one, and the entire EAI_AGAIN column vanishes with it — because the queries that overloaded the resolver never happen. The socket cap does the rest: even on a cold cache, connections: 64 bounds first-contact lookups to 64 instead of 200.

Why Cap the TTL at 60 Seconds

Cacheable-lookup honors each record’s TTL by default, and production CDNs routinely serve TTLs of five minutes to hours. During a multi-hour backfill, an uncapped cache pins you to one edge IP long after the provider may have rotated it away. maxTtl bounds that staleness — but check the units before you ship, because the option is milliseconds, not seconds. maxTtl: 60 caps the cache at sixty milliseconds, which is a no-op dressed up as a fix; the one-minute cap this post promises is maxTtl: 60_000. If the site you scrape rotates IPs aggressively — some bot walls do this deliberately — tighten it to 30_000. One more caveat: failed lookups aren’t cached in any useful sense — errored entries are discarded almost immediately — so a saturated resolver still stings on every fresh window. That’s the case for Fix Two.

Fix Two: Bound the Fan-Out, Not Just the Socket Count

The cache kills redundant lookups, but first contact with a new host still fires one lookup per socket, and connections: 64 means up to 64 of them can land at once. Your burst experiment already showed the resolver chokes well below that — the limiter-of-5 rerun was clean. So queue the backfill itself:

// limit the backfill — the Agent cap bounds sockets, not requests in flight
import pLimit from 'p-limit';

const limit = pLimit(10);                  // start from your probe numbers, not this post

await Promise.all(
  Array.from({ length: 200 }, (_, i) => limit(() => hit(i + 1)))
);

Rerun the burst through the limiter with the cache in place:

$ node burst.mjs                # cached + limited
dns lookups: 1
done in 11.9s — ok=200, p50=780ms, p99=1600ms

Be honest about that wall clock: 11.9s is slower than the 4.1s cached-and-unlimited run, though still 3.5× faster than the broken burst you started with. That’s the trade, and it’s usually the right one. The unlimited run only stayed clean because the cache collapsed lookups to one — but a cold cache, a new subdomain, or a TTL expiry mid-burst each fire connections-worth of simultaneous lookups at a resolver your probe showed choking well below that. The limiter is the seatbelt for exactly the cases the cache doesn’t cover.

Pick the Limit from the Probe, Not from This Post

dns-probe.mjs already bracketed the answer for you: somewhere between the 1-per-second run (clean) and the 100-in-flight run (saturated) is your resolver’s real ceiling. Start the limiter low, watch p99, and creep upward until the tail bends — then back off 20%. Keep the limiter even after Fix Three; burst tolerance on someone else’s resolver is a property they can revoke without telling you.

Fix Three: When dig Says the Resolver Itself Is the Problem

If the dig cross-check degraded identically under parallel load, no Node-side change fully saves you — you’re shipping queries to a resolver with a hard ceiling. Two environment fixes, in order of effort.

In Docker: Route Around the Embedded DNS

Docker’s embedded server at 127.0.0.11 is the usual offender — a low concurrent-transaction limit applied per container. Point the scraper service at a public resolver instead:

# docker-compose.yml — bypass the embedded DNS ceiling
services:
  scraper:
    image: scraper:latest
    dns:
      - 1.1.1.1
      - 8.8.8.8

The trade-off: you lose container-name resolution, which a scraper that only talks to the outside world doesn’t need. Pair this with Fix One so the public resolver sees one query per TTL window, not two hundred.

On a Host: Absorb Bursts Locally

If you crawl from a laptop or VPS behind a home router, install a caching resolver — unbound or dnsmasq — and point /etc/resolv.conf at 127.0.0.1. Your bursts land in a local cache with no per-client rate limit instead of being forwarded two hundred at a time to an underpowered upstream. Recheck with dns-probe.mjs afterward: the in-flight numbers should flatten to near the 1-per-second baseline.

Close the Loop: Rerun the Burst You Started With

The diagnosis script is also the acceptance test. This is the same 200-request burst after all three fixes — cache attached, limiter at 10, and the environment repaired per Fix Three (the container now pointed at 1.1.1.1, or unbound on 127.0.0.1 for a host deployment):

$ node burst.mjs
dns lookups: 2
done in 6.8s — ok=200, p50=430ms, p99=1500ms

Two lookups, not one: the cold start plus a single TTL refresh when the site’s own short record TTL expired mid-run — exactly the behavior you configured. Zero error codes, and a wall clock that beats the burst that opened this post by six times. If you see anything else, the measurement loop from earlier still applies — probe DNS alone, cross-check with dig, and fix the layer that’s actually bending. Symptom, reproduce, isolate, fix, verify: the loop is closed.

The reflex fixes from the intro can retire now. Cranking UV_THREADPOOL_SIZE relieves the local queue while doubling pressure on the resolver. Pinning /etc/hosts freezes an IP the CDN may rotate out tonight. A blind retry loop is the amplifier that turned a saturated resolver into those ENOTFOUND lies. Cache the resolution, bound the fan-out, and give the process a resolver that can take it — in that order, measured at each step.

#dns #networking #web-scraping #scraping-operations #troubleshooting #slot:operations

Related Articles