Skip to content
Technical 13 min read

Count the Requests Behind Every Rendered Page

Log every subrequest a rendered page fires with a local network listener. Measure request fan-out per URL, spot bloat, and find hidden data endpoints.

FE
FineData Engineering · Editorial Policy
|
On this page

Why a Single Page Load Is Never a Single Request

Ask anyone what happens when a browser loads a page and you’ll get the same answer: it fetches the HTML. One request. Maybe two if there’s a redirect. That mental model is fine for a static site from two decades ago. It is completely wrong for anything rendered today.

A modern page load is a fan-out. The HTML arrives, the parser starts pulling scripts, the scripts pull chunks, fonts, and images, and half of them fire XHR calls to API hosts you never typed into the URL bar. The page you see at example.com is assembled from thirty or forty network round-trips, and the URL bar tells you about exactly one of them.

Here’s the gap between expectation and reality for a single load of example.com:

What you expectWhat actually happensRequests
GET example.com HTMLHTML document1
—CSS bundles3
—JS bundles (app, vendor, runtime)7
—Font files (woff2)4
—XHR to store.example.com/api/cart2
—XHR to store.example.com/api/session6
—Tracking pixels and beacons5
—Prefetched next-route chunks2
Total expected: 1Total observed: 28+

If you’re scraping, this fan-out matters for three reasons. First, cost: every rendered page you pay for is actually dozens of network transactions, and render time is dominated by the slowest of them. Second, latency: you cannot know when a page is “done” without knowing how many requests it fires. Third, and this is the one people undervalue — the subrequest stream is a map of the site’s real API. The data you’re trying to scrape often flows through one of those XHR calls, in a cleaner shape than anything in the DOM.

View-source doesn’t show you any of this. DevTools does, but DevTools doesn’t scale to a pipeline, and it doesn’t produce a log you can diff. So we build a listener.

Setting Up a Local Network Listener That Records Every Subrequest

The tool we need is a local proxy that sits between the browser and the internet, logging every request that passes through. You could hand-roll a socket-level proxy, but you’d be re-implementing TLS interception, certificate generation, and HTTP/2 framing from scratch. Don’t. Use mitmproxy — it handles the TLS machinery, and you write a small addon that just records what flows past.

# fanout_addon.py
import json
import time

LOG_PATH = "/tmp/fanout.jsonl"

def request(flow):
    """Called for every request mitmproxy sees, in order."""
    record = {
        "ts": time.time(),
        "method": flow.request.method,
        "host": flow.request.pretty_host,
        "path": flow.request.path,
        "content_type": flow.request.headers.get("accept", ""),
    }
    with open(LOG_PATH, "a") as f:
        f.write(json.dumps(record) + "\n")

Run it on port 8080:

mitmdump -s fanout_addon.py -p 8080 --set flow_detail=0

flow_detail=0 keeps the console quiet — you want the JSONL file, not a wall of terminal output. Each line in that file is one request, with method, host, path, and a timestamp. That timestamp is what lets you reconstruct ordering later, which matters when you’re hunting polling loops.

Because mitmproxy intercepts TLS, the browser will see a certificate it doesn’t trust. For a local measurement harness that’s fine — you launch the browser with certificate errors ignored. Never do this in production traffic, but here you’re rendering example.com in a throwaway container and reading the log. The risk surface is a temp directory.

The config to point a browser at the listener:

PROXY = {"server": "http://127.0.0.1:8080"}

Playwright accepts this at launch or context level. One gotcha: some browsers route a few requests around the proxy (OCSP checks, certain extensions). For Chromium in a clean profile, the capture is effectively complete. If you need to prove completeness, compare the listener count against the browser’s own request count for one page — they should match within one or two.

Driving a Real Page Through the Listener with a Headless Browser

A proxy alone captures nothing. You need something to generate traffic, and the traffic has to include everything JavaScript fires — not just the initial navigation. That means a real browser, scripted.

# render.py
import json
from playwright.sync_api import sync_playwright

TARGET = "https://example.com/products"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={"server": "http://127.0.0.1:8080"},
    )
    context = browser.new_context(ignore_https_errors=True)
    page = context.new_page()

    page.goto(TARGET, wait_until="networkidle", timeout=60_000)
    page.wait_for_timeout(2000)  # settle buffer for late beacons

    context.close()
    browser.close()

The ignore_https_errors=True on the context is what makes the mitmproxy certificate acceptable. The wait_until="networkidle" waits until no network connections have been open for 500ms — the same heuristic Playwright uses everywhere.

That heuristic is also the biggest measurement trap in this whole setup. Under-wait and you undercount the fan-out. I’ve seen it go like this on example.com:

Wait strategySubrequests captured
Fixed 3-second delay21
networkidle + 2s buffer34

The 3-second delay misses thirteen requests. Why? Because the page’s analytics bundle loads late, and it fires a chain of beacons after the initial paint. A fixed delay is a guess; networkidle is a signal. But networkidle alone can fire too early on pages that poll — the 2-second buffer catches the first poll cycle. If you want to go deeper on when a page is actually ready to extract from, I wrote about measuring DOM settle time separately.

One more thing: networkidle will hang forever on pages that never stop polling. Cap it with a timeout and fall back to a fixed wait plus a manual “did the data selector appear” check.

Parsing the Request Log into a Per-URL Fan-Out Report

A flat JSONL log of 34 requests is noise. What you want is structure: how many requests per host, per path, and which paths get hit repeatedly.

# parse_fanout.py
import json
from collections import Counter
from urllib.parse import urlsplit

LOG_PATH = "/tmp/fanout.jsonl"
DUPLICATE_THRESHOLD = 3

def classify(path: str, accept: str) -> str:
    ext = path.split("?")[0].rsplit(".", 1)[-1].lower()
    if ext in ("js", "mjs"):
        return "script"
    if ext == "css":
        return "stylesheet"
    if ext in ("woff2", "woff", "ttf"):
        return "font"
    if ext in ("png", "jpg", "jpeg", "webp", "gif", "svg", "avif"):
        return "image"
    if "json" in accept or ext == "json":
        return "xhr"
    if path.endswith("/") or ext == "html" or "." not in path.rsplit("/", 1)[-1]:
        return "document"
    return "other"

def build_report(log_path: str) -> list[dict]:
    rows = []
    with open(log_path) as f:
        for line in f:
            r = json.loads(line)
            rows.append({
                "host": r["host"],
                "path": urlsplit("?" + r["path"]).path
                        if r["path"].startswith("?") else r["path"].split("?")[0],
                "type": classify(r["path"], r["content_type"]),
            })

    counts = Counter((r["host"], r["path"]) for r in rows)
    types = {(r["host"], r["path"]): r["type"] for r in rows}

    report = []
    for (host, path), count in counts.most_common():
        report.append({
            "host": host,
            "path": path,
            "count": count,
            "type": types[(host, path)],
            "flagged": count > DUPLICATE_THRESHOLD,
        })
    return report

if __name__ == "__main__":
    for row in build_report(LOG_PATH):
        marker = "  <-- duplicate fan-out" if row["flagged"] else ""
        print(f"{row['count']:>3}x  {row['type']:<10} {row['host']}{row['path']}{marker}")

Run it against the capture from example.com/products and you get something like this:

CountTypeHostPath
1documentexample.com/products
6xhrstore.example.com/api/session
3xhrstore.example.com/api/cart
2scriptstatic.example.com/assets/analytics.js
2scriptstatic.example.com/assets/app.js
4fontstatic.example.com/assets/fonts/inter.woff2
1scriptstatic.example.com/assets/vendor.js
5imagecdn.example.com/img/product/*.jpg
2xhrstore.example.com/api/internal/pricing

The flagged field is doing real work here. Any path hit more than three times per page load is either a polling loop, a retry storm, or a genuinely chatty endpoint — all three are worth knowing about before you build a pipeline that renders this page a thousand times a day.

Spotting Bloat: Duplicate Scripts, Redundant API Calls, and Tracking Pixels

The fan-out report is where waste becomes visible. Here’s a raw excerpt from the example.com capture, annotated:

14:02:01.102  GET  static.example.com/assets/analytics.js      # bundled copy
14:02:01.118  GET  cdn-partner.example.com/analytics.js        # same lib, CDN copy
14:02:01.840  GET  store.example.com/api/poll                  # t=0
14:02:03.851  GET  store.example.com/api/poll                  # t=+2s
14:02:05.862  GET  store.example.com/api/poll                  # t=+4s
14:02:07.870  GET  store.example.com/api/poll                  # +6s... still going

Two problems jump out. analytics.js is loaded twice — once bundled by the site, once from a partner CDN. That’s roughly 45 KB of duplicated script plus an extra DNS lookup and TLS handshake on every page load. And /api/poll fires every two seconds, indefinitely. In a scraping context this is worse than waste: it means your page never reaches networkidle, your waits time out, and your per-page render cost balloons because you’re holding a browser alive while it polls a cart status nobody asked for.

Now the uncomfortable opinion: most of this fan-out is none of your business. If you’re scraping example.com for product data, you do not need the fonts, the images, or any of the tracking pixels. They’re often 70-80% of the bytes transferred per load. Blocking them changes nothing about the data you extract and cuts your render time dramatically.

Here’s the before/after for the same page, with fonts, images, and known tracker hosts blocked at the route level:

MetricFull fan-outAfter dedup/block
Requests per page load3412
Estimated payload~2.1 MB~340 KB
Time to data present in DOM4.8s2.2s

The trade-off to be honest about: aggressive blocking can break pages that lazily hydrate components on image load events, or that gate content behind an analytics call. Block, then re-verify your extraction selectors against the blocked render. If your data still lands in the DOM, the block list is safe. If it doesn’t, unblock one host at a time until it does. For more on the rendering-cost side of this equation, see JS rendering vs plain HTTP — the per-request economics are the same whether you run the browser or an API does.

Finding Hidden Data Endpoints in the Subrequest Stream

Here’s where the listener stops being a performance tool and becomes a discovery tool. Buried in that fan-out log is the site’s real API — the endpoints the frontend calls to fill the page with data. These are frequently undocumented, unversioned, and far more pleasant to consume than the rendered DOM.

Filter the log for XHR-shaped requests:

# find_endpoints.py
import json

LOG_PATH = "/tmp/fanout.jsonl"

with open(LOG_PATH) as f:
    for line in f:
        r = json.loads(line)
        looks_json = "json" in r["content_type"].lower()
        looks_api = "/api/" in r["path"]
        if looks_json or looks_api:
            print(f"{r['method']:>4}  https://{r['host']}{r['path']}")

Output from the example.com/products capture:

 GET  https://store.example.com/api/session
 GET  https://store.example.com/api/cart
 GET  https://store.example.com/api/internal/pricing?region=eu&currency=EUR
POST  https://store.example.com/api/events

That third line is the interesting one. /api/internal/pricing with region=eu — the frontend is fetching localized pricing as structured JSON. That’s exactly the data a naive scraper would try to claw out of the DOM with selector chains, and here it is, pre-parsed, with parameters that tell you how to get every region. This is the same argument I make in hidden JSON endpoints vs DOM parsing: the subrequest stream usually beats the DOM.

Replay it to inspect the response shape:

curl -s "https://store.example.com/api/internal/pricing?region=eu&currency=EUR" \
  -H "Accept: application/json" | jq 'keys'

If the endpoint is open, that’s all you need. If it’s behind anti-bot checks or geo-locked, you can route the replay through a scraping API instead — a single POST /api/v1/scrape call with the discovered URL, proxy_country: "DE" for the EU exit, and raw_output: true with a single format gets you the raw JSON body back without rendering anything:

curl -s -X POST https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer fd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/api/internal/pricing?region=eu&currency=EUR",
    "proxy_country": "DE",
    "formats": ["text"],
    "raw_output": true,
    "use_antibot": true
  }'

One endpoint discovered, one request per region, zero browser. That is the payoff for logging the fan-out.

Automating Fan-Out Checks into a Repeatable Measurement

A one-off capture is a curiosity. A check that runs on every change to your target site — or every change to your own block list — is an engineering control. Wrap the three pieces (listener, renderer, parser) into one script with a pass/fail threshold.

First the config:

{
  "listener_port": 8080,
  "wait_strategy": "networkidle",
  "settle_buffer_ms": 2000,
  "targets": [
    {
      "url": "https://example.com",
      "max_requests": 40
    },
    {
      "url": "https://store.example.com/catalog",
      "max_requests": 60
    }
  ]
}

Then the runner:

# fanout_check.py
import json
import subprocess
import sys
import time
from pathlib import Path

from playwright.sync_api import sync_playwright

CONFIG = json.loads(Path("fanout_config.json").read_text())
LOG = Path("/tmp/fanout.jsonl")

def start_listener(port: int) -> subprocess.Popen:
    LOG.write_text("")  # truncate per run
    return subprocess.Popen(
        ["mitmdump", "-s", "fanout_addon.py",
         "-p", str(port), "--set", "flow_detail=0"],
        stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
    )

def render(target: dict, port: int) -> int:
    before = LOG.stat().st_size if LOG.exists() else 0
    with sync_playwright() as p:
        browser = p.chromium.launch(
            headless=True,
            proxy={"server": f"http://127.0.0.1:{port}"},
        )
        context = browser.new_context(ignore_https_errors=True)
        page = context.new_page()
        try:
            page.goto(target["url"],
                     wait_until=CONFIG["wait_strategy"],
                     timeout=60_000)
        except Exception:
            print(f"  networkidle timed out (polling page?), "
                  f"counting what we got", file=sys.stderr)
        page.wait_for_timeout(CONFIG["settle_buffer_ms"])
        context.close()
        browser.close()
    lines = LOG.read_text().splitlines()
    return len([l for l in lines if l.strip()])

def main() -> int:
    proxy = start_listener(CONFIG["listener_port"])
    time.sleep(2)  # let mitmdump bind

    failures = []
    try:
        for target in CONFIG["targets"]:
            count = render(target, CONFIG["listener_port"])
            status = "OK " if count <= target["max_requests"] else "FAIL"
            print(f"[{status}] {target['url']}: {count} subrequests "
                  f"(budget {target['max_requests']})")
            if count > target["max_requests"]:
                failures.append(target["url"])
    finally:
        proxy.terminate()

    if failures:
        print(f"\n{len(failures)} target(s) over budget — "
              f"check for new trackers or polling loops.")
        return 1
    return 0

if __name__ == "__main__":
    sys.exit(main())

Exit code 1 on failure means this drops straight into CI. When a site redesign adds a new tag manager or a chat widget that polls every second, your pipeline fails loudly instead of silently doubling its render cost.

Two practical notes. The per-run truncation of the log matters — without it, counts accumulate across targets and your budgets are fiction. And the networkidle timeout path is not a bug: polling pages will never go idle, so count what fired and flag it in the report rather than crashing the check.

Wrap-Up

A rendered page is never one request, and pretending otherwise costs you money, latency, and visibility into where the data actually lives. The stack here is deliberately small: a mitmproxy addon that logs every subrequest, a scripted browser to generate real traffic, a parser that turns the log into a per-host, per-path fan-out report, and a threshold check that runs on demand or in CI.

The three payoffs, in order of how often they matter to me: blocking the 70-80% of fan-out you don’t need, catching polling loops that break your wait strategies, and discovering JSON endpoints that let you skip the browser entirely. The last one is the highest-leverage — a single discovered endpoint can replace an entire DOM-scraping path.

Start with one target URL and a generous budget. Tighten it as you learn what the page actually needs. The numbers you’ll gather along the way — requests per page, bytes per load, which hosts repeat — feed directly into cost forecasting and refresh scheduling, which is a topic for another day.

#playwright #network tracing #request fan-out #web scraping #data engineering #slot:measurement

Related Articles