Count the Requests Behind Every Rendered Page
Log every subrequest a rendered page fires with a local network listener. Measure request fan-out per URL, spot bloat, and find hidden data endpoints.
On this page
Why a Single Page Load Is Never a Single Request
Ask anyone what happens when a browser loads a page and you’ll get the same answer: it fetches the HTML. One request. Maybe two if there’s a redirect. That mental model is fine for a static site from two decades ago. It is completely wrong for anything rendered today.
A modern page load is a fan-out. The HTML arrives, the parser starts pulling scripts, the scripts pull chunks, fonts, and images, and half of them fire XHR calls to API hosts you never typed into the URL bar. The page you see at example.com is assembled from thirty or forty network round-trips, and the URL bar tells you about exactly one of them.
Here’s the gap between expectation and reality for a single load of example.com:
| What you expect | What actually happens | Requests |
|---|---|---|
GET example.com HTML | HTML document | 1 |
| — | CSS bundles | 3 |
| — | JS bundles (app, vendor, runtime) | 7 |
| — | Font files (woff2) | 4 |
| — | XHR to store.example.com/api/cart | 2 |
| — | XHR to store.example.com/api/session | 6 |
| — | Tracking pixels and beacons | 5 |
| — | Prefetched next-route chunks | 2 |
| Total expected: 1 | Total observed: 28+ |
If you’re scraping, this fan-out matters for three reasons. First, cost: every rendered page you pay for is actually dozens of network transactions, and render time is dominated by the slowest of them. Second, latency: you cannot know when a page is “done” without knowing how many requests it fires. Third, and this is the one people undervalue — the subrequest stream is a map of the site’s real API. The data you’re trying to scrape often flows through one of those XHR calls, in a cleaner shape than anything in the DOM.
View-source doesn’t show you any of this. DevTools does, but DevTools doesn’t scale to a pipeline, and it doesn’t produce a log you can diff. So we build a listener.
Setting Up a Local Network Listener That Records Every Subrequest
The tool we need is a local proxy that sits between the browser and the internet, logging every request that passes through. You could hand-roll a socket-level proxy, but you’d be re-implementing TLS interception, certificate generation, and HTTP/2 framing from scratch. Don’t. Use mitmproxy — it handles the TLS machinery, and you write a small addon that just records what flows past.
# fanout_addon.py
import json
import time
LOG_PATH = "/tmp/fanout.jsonl"
def request(flow):
"""Called for every request mitmproxy sees, in order."""
record = {
"ts": time.time(),
"method": flow.request.method,
"host": flow.request.pretty_host,
"path": flow.request.path,
"content_type": flow.request.headers.get("accept", ""),
}
with open(LOG_PATH, "a") as f:
f.write(json.dumps(record) + "\n")
Run it on port 8080:
mitmdump -s fanout_addon.py -p 8080 --set flow_detail=0
flow_detail=0 keeps the console quiet — you want the JSONL file, not a wall of terminal output. Each line in that file is one request, with method, host, path, and a timestamp. That timestamp is what lets you reconstruct ordering later, which matters when you’re hunting polling loops.
Because mitmproxy intercepts TLS, the browser will see a certificate it doesn’t trust. For a local measurement harness that’s fine — you launch the browser with certificate errors ignored. Never do this in production traffic, but here you’re rendering example.com in a throwaway container and reading the log. The risk surface is a temp directory.
The config to point a browser at the listener:
PROXY = {"server": "http://127.0.0.1:8080"}
Playwright accepts this at launch or context level. One gotcha: some browsers route a few requests around the proxy (OCSP checks, certain extensions). For Chromium in a clean profile, the capture is effectively complete. If you need to prove completeness, compare the listener count against the browser’s own request count for one page — they should match within one or two.
Driving a Real Page Through the Listener with a Headless Browser
A proxy alone captures nothing. You need something to generate traffic, and the traffic has to include everything JavaScript fires — not just the initial navigation. That means a real browser, scripted.
# render.py
import json
from playwright.sync_api import sync_playwright
TARGET = "https://example.com/products"
with sync_playwright() as p:
browser = p.chromium.launch(
headless=True,
proxy={"server": "http://127.0.0.1:8080"},
)
context = browser.new_context(ignore_https_errors=True)
page = context.new_page()
page.goto(TARGET, wait_until="networkidle", timeout=60_000)
page.wait_for_timeout(2000) # settle buffer for late beacons
context.close()
browser.close()
The ignore_https_errors=True on the context is what makes the mitmproxy certificate acceptable. The wait_until="networkidle" waits until no network connections have been open for 500ms — the same heuristic Playwright uses everywhere.
That heuristic is also the biggest measurement trap in this whole setup. Under-wait and you undercount the fan-out. I’ve seen it go like this on example.com:
| Wait strategy | Subrequests captured |
|---|---|
| Fixed 3-second delay | 21 |
networkidle + 2s buffer | 34 |
The 3-second delay misses thirteen requests. Why? Because the page’s analytics bundle loads late, and it fires a chain of beacons after the initial paint. A fixed delay is a guess; networkidle is a signal. But networkidle alone can fire too early on pages that poll — the 2-second buffer catches the first poll cycle. If you want to go deeper on when a page is actually ready to extract from, I wrote about measuring DOM settle time separately.
One more thing: networkidle will hang forever on pages that never stop polling. Cap it with a timeout and fall back to a fixed wait plus a manual “did the data selector appear” check.
Parsing the Request Log into a Per-URL Fan-Out Report
A flat JSONL log of 34 requests is noise. What you want is structure: how many requests per host, per path, and which paths get hit repeatedly.
# parse_fanout.py
import json
from collections import Counter
from urllib.parse import urlsplit
LOG_PATH = "/tmp/fanout.jsonl"
DUPLICATE_THRESHOLD = 3
def classify(path: str, accept: str) -> str:
ext = path.split("?")[0].rsplit(".", 1)[-1].lower()
if ext in ("js", "mjs"):
return "script"
if ext == "css":
return "stylesheet"
if ext in ("woff2", "woff", "ttf"):
return "font"
if ext in ("png", "jpg", "jpeg", "webp", "gif", "svg", "avif"):
return "image"
if "json" in accept or ext == "json":
return "xhr"
if path.endswith("/") or ext == "html" or "." not in path.rsplit("/", 1)[-1]:
return "document"
return "other"
def build_report(log_path: str) -> list[dict]:
rows = []
with open(log_path) as f:
for line in f:
r = json.loads(line)
rows.append({
"host": r["host"],
"path": urlsplit("?" + r["path"]).path
if r["path"].startswith("?") else r["path"].split("?")[0],
"type": classify(r["path"], r["content_type"]),
})
counts = Counter((r["host"], r["path"]) for r in rows)
types = {(r["host"], r["path"]): r["type"] for r in rows}
report = []
for (host, path), count in counts.most_common():
report.append({
"host": host,
"path": path,
"count": count,
"type": types[(host, path)],
"flagged": count > DUPLICATE_THRESHOLD,
})
return report
if __name__ == "__main__":
for row in build_report(LOG_PATH):
marker = " <-- duplicate fan-out" if row["flagged"] else ""
print(f"{row['count']:>3}x {row['type']:<10} {row['host']}{row['path']}{marker}")
Run it against the capture from example.com/products and you get something like this:
| Count | Type | Host | Path |
|---|---|---|---|
| 1 | document | example.com | /products |
| 6 | xhr | store.example.com | /api/session |
| 3 | xhr | store.example.com | /api/cart |
| 2 | script | static.example.com | /assets/analytics.js |
| 2 | script | static.example.com | /assets/app.js |
| 4 | font | static.example.com | /assets/fonts/inter.woff2 |
| 1 | script | static.example.com | /assets/vendor.js |
| 5 | image | cdn.example.com | /img/product/*.jpg |
| 2 | xhr | store.example.com | /api/internal/pricing |
The flagged field is doing real work here. Any path hit more than three times per page load is either a polling loop, a retry storm, or a genuinely chatty endpoint — all three are worth knowing about before you build a pipeline that renders this page a thousand times a day.
Spotting Bloat: Duplicate Scripts, Redundant API Calls, and Tracking Pixels
The fan-out report is where waste becomes visible. Here’s a raw excerpt from the example.com capture, annotated:
14:02:01.102 GET static.example.com/assets/analytics.js # bundled copy
14:02:01.118 GET cdn-partner.example.com/analytics.js # same lib, CDN copy
14:02:01.840 GET store.example.com/api/poll # t=0
14:02:03.851 GET store.example.com/api/poll # t=+2s
14:02:05.862 GET store.example.com/api/poll # t=+4s
14:02:07.870 GET store.example.com/api/poll # +6s... still going
Two problems jump out. analytics.js is loaded twice — once bundled by the site, once from a partner CDN. That’s roughly 45 KB of duplicated script plus an extra DNS lookup and TLS handshake on every page load. And /api/poll fires every two seconds, indefinitely. In a scraping context this is worse than waste: it means your page never reaches networkidle, your waits time out, and your per-page render cost balloons because you’re holding a browser alive while it polls a cart status nobody asked for.
Now the uncomfortable opinion: most of this fan-out is none of your business. If you’re scraping example.com for product data, you do not need the fonts, the images, or any of the tracking pixels. They’re often 70-80% of the bytes transferred per load. Blocking them changes nothing about the data you extract and cuts your render time dramatically.
Here’s the before/after for the same page, with fonts, images, and known tracker hosts blocked at the route level:
| Metric | Full fan-out | After dedup/block |
|---|---|---|
| Requests per page load | 34 | 12 |
| Estimated payload | ~2.1 MB | ~340 KB |
| Time to data present in DOM | 4.8s | 2.2s |
The trade-off to be honest about: aggressive blocking can break pages that lazily hydrate components on image load events, or that gate content behind an analytics call. Block, then re-verify your extraction selectors against the blocked render. If your data still lands in the DOM, the block list is safe. If it doesn’t, unblock one host at a time until it does. For more on the rendering-cost side of this equation, see JS rendering vs plain HTTP — the per-request economics are the same whether you run the browser or an API does.
Finding Hidden Data Endpoints in the Subrequest Stream
Here’s where the listener stops being a performance tool and becomes a discovery tool. Buried in that fan-out log is the site’s real API — the endpoints the frontend calls to fill the page with data. These are frequently undocumented, unversioned, and far more pleasant to consume than the rendered DOM.
Filter the log for XHR-shaped requests:
# find_endpoints.py
import json
LOG_PATH = "/tmp/fanout.jsonl"
with open(LOG_PATH) as f:
for line in f:
r = json.loads(line)
looks_json = "json" in r["content_type"].lower()
looks_api = "/api/" in r["path"]
if looks_json or looks_api:
print(f"{r['method']:>4} https://{r['host']}{r['path']}")
Output from the example.com/products capture:
GET https://store.example.com/api/session
GET https://store.example.com/api/cart
GET https://store.example.com/api/internal/pricing?region=eu¤cy=EUR
POST https://store.example.com/api/events
That third line is the interesting one. /api/internal/pricing with region=eu — the frontend is fetching localized pricing as structured JSON. That’s exactly the data a naive scraper would try to claw out of the DOM with selector chains, and here it is, pre-parsed, with parameters that tell you how to get every region. This is the same argument I make in hidden JSON endpoints vs DOM parsing: the subrequest stream usually beats the DOM.
Replay it to inspect the response shape:
curl -s "https://store.example.com/api/internal/pricing?region=eu¤cy=EUR" \
-H "Accept: application/json" | jq 'keys'
If the endpoint is open, that’s all you need. If it’s behind anti-bot checks or geo-locked, you can route the replay through a scraping API instead — a single POST /api/v1/scrape call with the discovered URL, proxy_country: "DE" for the EU exit, and raw_output: true with a single format gets you the raw JSON body back without rendering anything:
curl -s -X POST https://api.finedata.ai/api/v1/scrape \
-H "Authorization: Bearer fd_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://store.example.com/api/internal/pricing?region=eu¤cy=EUR",
"proxy_country": "DE",
"formats": ["text"],
"raw_output": true,
"use_antibot": true
}'
One endpoint discovered, one request per region, zero browser. That is the payoff for logging the fan-out.
Automating Fan-Out Checks into a Repeatable Measurement
A one-off capture is a curiosity. A check that runs on every change to your target site — or every change to your own block list — is an engineering control. Wrap the three pieces (listener, renderer, parser) into one script with a pass/fail threshold.
First the config:
{
"listener_port": 8080,
"wait_strategy": "networkidle",
"settle_buffer_ms": 2000,
"targets": [
{
"url": "https://example.com",
"max_requests": 40
},
{
"url": "https://store.example.com/catalog",
"max_requests": 60
}
]
}
Then the runner:
# fanout_check.py
import json
import subprocess
import sys
import time
from pathlib import Path
from playwright.sync_api import sync_playwright
CONFIG = json.loads(Path("fanout_config.json").read_text())
LOG = Path("/tmp/fanout.jsonl")
def start_listener(port: int) -> subprocess.Popen:
LOG.write_text("") # truncate per run
return subprocess.Popen(
["mitmdump", "-s", "fanout_addon.py",
"-p", str(port), "--set", "flow_detail=0"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
)
def render(target: dict, port: int) -> int:
before = LOG.stat().st_size if LOG.exists() else 0
with sync_playwright() as p:
browser = p.chromium.launch(
headless=True,
proxy={"server": f"http://127.0.0.1:{port}"},
)
context = browser.new_context(ignore_https_errors=True)
page = context.new_page()
try:
page.goto(target["url"],
wait_until=CONFIG["wait_strategy"],
timeout=60_000)
except Exception:
print(f" networkidle timed out (polling page?), "
f"counting what we got", file=sys.stderr)
page.wait_for_timeout(CONFIG["settle_buffer_ms"])
context.close()
browser.close()
lines = LOG.read_text().splitlines()
return len([l for l in lines if l.strip()])
def main() -> int:
proxy = start_listener(CONFIG["listener_port"])
time.sleep(2) # let mitmdump bind
failures = []
try:
for target in CONFIG["targets"]:
count = render(target, CONFIG["listener_port"])
status = "OK " if count <= target["max_requests"] else "FAIL"
print(f"[{status}] {target['url']}: {count} subrequests "
f"(budget {target['max_requests']})")
if count > target["max_requests"]:
failures.append(target["url"])
finally:
proxy.terminate()
if failures:
print(f"\n{len(failures)} target(s) over budget — "
f"check for new trackers or polling loops.")
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
Exit code 1 on failure means this drops straight into CI. When a site redesign adds a new tag manager or a chat widget that polls every second, your pipeline fails loudly instead of silently doubling its render cost.
Two practical notes. The per-run truncation of the log matters — without it, counts accumulate across targets and your budgets are fiction. And the networkidle timeout path is not a bug: polling pages will never go idle, so count what fired and flag it in the report rather than crashing the check.
Wrap-Up
A rendered page is never one request, and pretending otherwise costs you money, latency, and visibility into where the data actually lives. The stack here is deliberately small: a mitmproxy addon that logs every subrequest, a scripted browser to generate real traffic, a parser that turns the log into a per-host, per-path fan-out report, and a threshold check that runs on demand or in CI.
The three payoffs, in order of how often they matter to me: blocking the 70-80% of fan-out you don’t need, catching polling loops that break your wait strategies, and discovering JSON endpoints that let you skip the browser entirely. The last one is the highest-leverage — a single discovered endpoint can replace an entire DOM-scraping path.
Start with one target URL and a generous budget. Tighten it as you learn what the page actually needs. The numbers you’ll gather along the way — requests per page, bytes per load, which hosts repeat — feed directly into cost forecasting and refresh scheduling, which is a topic for another day.
Related Articles
Prove Your Concurrency: Count In-Flight Requests Locally
Point your scraper at a local listener that logs request starts and ends, then compute real in-flight concurrency to verify your worker pool setup.
TechnicalMeasure DOM Settle Time: Know When a Page Is Ready
Build a local MutationObserver harness that measures DOM quiescence on JavaScript-heavy pages, so you extract only after rendering truly settles.
TechnicalWhat a Proxy Hour Really Costs: A Burn Budget
Model the real cost of proxying a scrape: bandwidth per page, rotation overhead, and failure burn you can recompute from your own request logs.