Budget the Fetches Behind Every Data Row
Every data row rides on listing pages, pagination, and dead links. Model the fetch-to-row ratio, plug in real numbers, and budget crawls you can defend.
On this page
Introduction: The Number That’s Wrong Twice
Ask an engineer how many requests their next crawl will need and you usually get a row count with a shrug: “about 12,000, one per product.” That number is wrong twice. It ignores the listing and pagination pages that surface those products, and it assumes every fetched URL actually returns a row. The fix is a single measured ratio — fetches per extracted row — and once you have it, crawl budgets, politeness ceilings, and cost projections all collapse into arithmetic you can defend in a review.
Counting the Fetches Behind One Row: The Anatomy of the Ratio
Follow one product row on store.example.com from discovery to extraction. The URL trail looks like this:
1. https://store.example.com/c/tools # category landing, 48 product links
2. https://store.example.com/c/tools?page=2 # pagination, 44 links, 2 seen before
3. https://store.example.com/p/comb-drill-18v # detail page -> THE row
Three fetches touched this row’s life, but only one produced it. The listing fetch is shared across 48 rows; the pagination fetch across 44; the detail fetch is all yours. That’s the whole idea behind the fetch-to-row ratio: total fetches divided by distinct rows extracted. Listing fetches amortize. Detail fetches don’t.
Map every URL you touch into a class and the ratio becomes mechanical:
| URL class | Example | Rows per fetch | Fetches contributed per row |
|---|---|---|---|
| Category listing | /c/tools | 48 | 0.021 |
| Pagination page | /c/tools?page=7 | ~44 (decays deeper) | ~0.023 |
| Detail page | /p/comb-drill-18v | 1 | 1.0 |
| Redirect hop | /go/comb-drill-18v | 0 | 1.0 when hit |
| Dead link | /p/discontinued-sander | 0 | overhead only |
Sum the column and you get a floor of roughly 1.02 fetches per row before anything goes wrong. Redirects and dead links only push it up.
Classify as you crawl and keep a running ledger:
import re
from collections import Counter
from dataclasses import dataclass, field
CLASS_PATTERNS = [
("pagination", re.compile(r"store\.example\.com/c/[^?]+\?page=\d+")),
("listing", re.compile(r"store\.example\.com/c/[\w-]+$")),
("detail", re.compile(r"store\.example\.com/p/[\w-]+")),
("redirect", re.compile(r"store\.example\.com/go/")),
]
def classify(url: str) -> str:
for label, pattern in CLASS_PATTERNS:
if pattern.search(url):
return label
return "other"
@dataclass
class FetchLedger:
fetches: Counter = field(default_factory=Counter)
rows: int = 0
def record(self, url: str, rows_extracted: int):
self.fetches[classify(url)] += 1
self.rows += rows_extracted
@property
def ratio(self) -> float:
total = sum(self.fetches.values())
return round(total / self.rows, 3) if self.rows else float("inf")
The ratio property is the number worth watching. It should hover near 1.05 on a healthy crawl. When it starts drifting toward 1.3, something structural changed — dead links, duplicates, or a listing that stopped listing.
Pagination Styles That Multiply Fetches: Offset, Cursor, and Infinite Scroll
The same catalog can expose three different pagination schemes, and they do not cost the same per row. Offset pagination (?page=N) re-serves pages from the top; if products get inserted mid-crawl, page 2 becomes yesterday’s page 1 and you re-fetch rows you already have. Cursor pagination (?after=token) hands you a stable position, so overlap is zero by construction. Infinite scroll is usually a JSON feed wearing a UI costume — replaying scroll events is the expensive way to reach it, calling the feed directly is the cheap way.
| Scheme | Pattern | Rows per fetch | Listing fetches per 100 rows | Overlap risk |
|---|---|---|---|---|
| Offset | ?page=N (48/page) | 48 | 2.1 | High when data shifts mid-crawl |
| Cursor | ?after=token | 48 | 2.1 | None by design |
| Infinite scroll, DOM replay | scroll event loads 24 | 24 per XHR + base render | 4.2 + 1 render | Medium |
| Infinite scroll, JSON direct | /api/c/tools?limit=100 | 100 | 1.0 | None if you page the feed |
Enumerate the patterns explicitly in your crawler config so the budget arithmetic has something concrete to chew on:
pagination:
site: store.example.com
patterns:
- "store.example.com/c/tools?page={n}" # offset, n = 1..40
- "store.example.com/c/tools?after={cursor}" # cursor, when available
json_feed:
url: "https://store.example.com/api/c/tools"
params: { limit: 100, offset: 0 }
max_pages: 40
Now measure instead of assuming. A 20-page sample crawl on this store shows rows-per-page decaying as pagination deepens — delisted products, repeated placements, thinner tail pages:
| Pages | Avg rows per page |
|---|---|
| 1–5 | 48 |
| 6–10 | 44 |
| 11–15 | 41 |
| 16–20 | 35 |
Effective density across the sample is 42 rows per page, not 48. If you budget at the page-1 number, you undercount listing fetches by about 14% and you never see it coming, because every one of those pages still returns a 200.
Here’s where I’ll save you a wasted sprint: pagination optimization barely moves the total budget. Listing fetches are a rounding error next to detail fetches — the sensitivity table later in this post proves it numerically. Find the JSON feed if it’s obvious, but don’t spend days reverse-engineering scroll behavior to shave two fetches off a 14,000-fetch crawl.
Dead Links and Soft 404s: The Fetches That Return Zero Rows
Catalogs rot. Products get delisted and their detail URLs keep circulating as dead links in listings, sitemaps, and your own historical data. Some fail loudly with a 404. Worse are soft 404s: pages that return HTTP 200 and a perfectly rendered “Product not found” template. If your extractor keys on HTTP status alone, you’ll count those as successes and your dataset will quietly fill with empty rows.
Log status and a page signature together:
SOFT_404_SIGNATURES = (
"Product not found",
"This item is no longer available",
"Page not found",
)
def classify_outcome(status: int, html: str) -> str:
if 300 <= status < 400:
return "redirect"
if status == 404:
return "hard_404"
if status == 200:
for sig in SOFT_404_SIGNATURES:
if sig in html:
return "soft_404"
return "ok"
return f"http_{status}"
One fetch flag decides whether classify_outcome ever sees a redirect at all: request with allow_redirects=False. If your client follows the 3xx automatically, you’re handed the target’s 200 and the hop class vanishes from the ledger — even though you paid for both fetches.
Run this over a 500-link sample from store.example.com and the zero-yield fetches become countable:
| Outcome | Count | Fetches incurred | Rows extracted |
|---|---|---|---|
| 200, genuine detail | 431 | 431 | 431 |
| 3xx redirect | 42 | 84 (hop + target) | 38 |
| 404 | 19 | 19 | 0 |
| Soft 404 (returns 200) | 8 | 8 | 0 |
| Total | 500 | 542 | 469 |
That sample yields 0.87 rows per fetch, and the dead-link rate — hard 404s plus soft 404s plus redirects that land on 404 — sits at 6.2%. Every one of those 31 dead links was a fetch somebody paid for.
Retries interact with this badly unless you split failure classes first. Transient failures (timeouts, 5xx) deserve a retry. Permanent ones don’t, and never will:
| Retry policy | What gets retried | Extra fetches per 1,000 target rows | Rows recovered |
|---|---|---|---|
| Nothing | — | 0 | ~20 lost to transient errors |
| Everything, once | all non-200 | 104 | 20 |
| 5xx/timeout only, once | transient | 23 | 20 |
| 5xx/timeout only, twice | transient | 24 | 20 |
Same rows recovered, four and a half times the overhead when you retry indiscriminately. Retrying a 404 is a donation to the site’s analytics. I wrote about how retries compound into blended page cost in How Retries Quietly Double Your Scraping Costs — the short version: retry budgets belong in the ledger, not in the defaults.
Duplicate Paths: When Several Fetches Resolve to the Same Row
The same product is reachable from /c/tools, /c/power-tools, and a clearance cross-listing. Fetch all three paths and you extract the same row twice, paying twice. The dedup key has to be a canonical URL, built before the fetch, not the row content after it.
from urllib.parse import urlsplit, parse_qsl, urlencode
STRIP_PREFIXES = ("utm_", "fbclid", "gclid")
STRIP_EXACT = {"sid"} # exact match only — 'sidebar' and 'side' must survive
def canonical_url(url: str) -> str:
parts = urlsplit(url.lower())
query = sorted(
(k, v) for k, v in parse_qsl(parts.query)
if not k.startswith(STRIP_PREFIXES) and k not in STRIP_EXACT
)
path = parts.path.rstrip("/") or "/"
base = f"{parts.scheme}://{parts.netloc}{path}"
# Append the query string only when something survives stripping —
# a trailing '?' mints '/p/x?', which is a different string than '/p/x' in a seen-set
return f"{base}?{urlencode(query)}" if query else base
Note the two details that make the canonical form trustworthy: sid is stripped on exact key match only, so legitimate parameters like sidebar survive, and the ? is dropped entirely when no parameters remain, so stripped URLs collapse to exactly one key instead of two.
One honest caveat: lowercasing the path can over-merge on hosts with case-sensitive URLs. For this retailer the paths are slugs and the trade is worth it — a handful of false collisions costs less than one duplicate fetch. Your site may differ; check before you ship it.
Measured on 2,000 detail fetches, duplicates cluster where you’d expect — cross-listed categories:
| Category | Fetches | Distinct rows | Duplicate rate |
|---|---|---|---|
| tools | 800 | 768 | 4.0% |
| outdoor | 700 | 665 | 5.0% |
| clearance | 500 | 450 | 10.0% |
| Total | 2,000 | 1,883 | 5.9% |
That 5.9% is pure overhead, and it’s controllable:
| Strategy | Detail fetches for 12,000 distinct rows | Wasted fetches |
|---|---|---|
| Naive: fetch every discovered link | ~12,740 | ~740 |
| Seen-URL set at enqueue time | ~12,050 | ~50 |
The seen-set never hits zero waste — in-flight races and normalization misses leak a fraction of a percent — but 690 fetches saved on one crawl is 690 fetches you don’t pay for, proxy, or wait on. Dedupe before the fetch, at enqueue time. Doing it after parsing means you already paid for the bytes.
Politeness Math: Turning Rate Limits Into a Rows-Per-Day Ceiling
Everything above tells you how many fetches a crawl needs. Politeness tells you how many fetches a day will tolerate. store.example.com asks for this in robots.txt:
User-agent: *
Crawl-delay: 2
Disallow: /cart
Disallow: /account
Disallow: /search?
Crawl-delay of 2 means one request every two seconds. Concurrency does not repeal that — it hides latency, it doesn’t multiply throughput, because the site’s limiter buckets by IP regardless of how many connections you hold open. Anyone telling you ten connections means ten times the rate is selling something. Configure it honestly:
import threading, time
class RateGate:
def __init__(self, min_interval: float = 2.0, concurrency: int = 2):
self.min_interval = min_interval
self.sem = threading.Semaphore(concurrency)
self.lock = threading.Lock()
self.last = 0.0
def acquire(self):
with self.lock:
wait = self.last + self.min_interval - time.monotonic()
if wait > 0:
time.sleep(wait)
self.last = time.monotonic()
self.sem.acquire()
def release(self):
self.sem.release()
Two connections, one fetch every two seconds, globally paced. Now the ceiling falls out of division:
| Step | Value |
|---|---|
| Politeness floor | 1 fetch / 2 s |
| Max fetches per day | 86,400 / 2 = 43,200 |
| Fetch-to-row ratio (computed in this post) | ~1.16 |
| Max rows per day | ~37,000 |
And run duration inverts the same way: the 13,946-fetch budget in the next section needs just under 8 hours at this pace. If someone needs it in two hours, the conversation is about politeness, not code.
A Worked Budget: 12,000 Rows From store.example.com With Real Numbers
Assemble the pieces. Listing density, duplicate overhead, dead-link rate, and a retry allowance for transient failures only:
def fetch_budget(rows: int, rows_per_page: int, duplicate_rate: float,
dead_link_rate: float, transient_rate: float = 0.0,
retries: int = 0) -> int:
listing_fetches = rows / rows_per_page
detail_fetches = rows * (1 + duplicate_rate)
base = listing_fetches + detail_fetches
with_retries = base * (1 + transient_rate * retries)
return int(with_retries / (1 - dead_link_rate))
total = fetch_budget(
rows=12_000,
rows_per_page=48,
duplicate_rate=0.06, # measured ~5.9%, round up
dead_link_rate=0.07, # measured 6.2% on the sample, pad it
transient_rate=0.02, # 5xx + timeouts
retries=1,
)
# 250 listing + 12,720 detail = 12,970 base
# x 1.02 retries = 13,229, / 0.93 = 14,224
| Input | Value | Fetches |
|---|---|---|
| Rows target | 12,000 | — |
| Listing pages at 48 rows/page | 250 pages | 250 |
| Detail fetches with 6% duplicate overhead | 12,000 × 1.06 | 12,720 |
| Subtotal | 12,970 | |
| Retries on 2% transient failures, once | × 1.02 | ~259 |
| Dead-link overhead (7%) | ÷ (1 − 0.07) | ~14,220 |
Call it 14,250 fetches and nobody will accuse you of false precision. If those fetches run as billed jobs through a scraping provider like FineData, multiply the count by your per-fetch price — add-on options such as residential routing or JS rendering move the unit cost, not the count, so the ratio stays valid either way.
Now the sensitivity check, because a single-point estimate is an opinion and a range is an argument:
| Rows per page \ dead-link rate | 4% | 7% | 10% |
|---|---|---|---|
| 43 | 13,541 | 13,978 | 14,443 |
| 48 | 13,510 | 13,946 | 14,411 |
| 53 | 13,486 | 13,921 | 14,385 |
Read the axes carefully. Swinging rows-per-page by ten (43 to 53) moves the total by about 60 fetches. Swinging dead-link rate by six points moves it by about 900. The entire pagination discussion earlier in this post was worth roughly 0.4% of the budget; the dead-link rate is worth 6%. Budget your engineering time accordingly: link hygiene and extraction of delisted products first, feed reverse-engineering last. For tying these fetches back to actual dollars per URL, the approach in Build a Scrape Cost Ledger drops straight into the numbers above.
Instrumenting the Crawler to Emit a Fetch-to-Row Report You Can Defend
A budget without a post-run report is a guess wearing a spreadsheet. Emit the ledger every run:
import csv, json
def emit_report(ledger: FetchLedger, run_id: str, estimated: int | None = None):
total = sum(ledger.fetches.values())
dead = ledger.fetches.get("hard_404", 0) + ledger.fetches.get("soft_404", 0)
payload = {
"run_id": run_id,
"total_fetches": total,
"fetches_by_class": dict(sorted(ledger.fetches.items())),
"rows_extracted": ledger.rows,
"fetch_to_row_ratio": round(total / ledger.rows, 3) if ledger.rows else None,
"dead_link_rate": round(dead / total, 4),
"estimated_fetches": estimated,
# guard with `is not None` — a falsy-but-real estimate must still report
"variance_pct": round((total - estimated) / estimated * 100, 1)
if estimated is not None else None,
}
with open(f"fetch_report_{run_id}.json", "w") as fh:
json.dump(payload, fh, indent=2)
with open(f"fetch_report_{run_id}.csv", "w", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(["class", "fetches"])
writer.writerows(ledger.fetches.items())
return payload
If the crawl runs through a scraping API, reconcile your local ledger against the provider’s counters so the two never drift apart silently. Before wiring that reconciliation in, prove the request path itself is healthy with the smallest probe that can fail loudly — a plain GET against example.com, the one scrape target every environment permits, wrapped so a 405 or a dropped connection prints a status instead of crashing the run:
import requests
try:
resp = requests.get("https://example.com", timeout=10)
# print the status so a 405/5xx shows up in the log instead of poisoning the ledger
print(resp.status_code)
except requests.RequestException as exc:
print(f"request failed: {exc}")
GET /api/v1/async/jobs?status=failed gives you the failure-side cross-check for the same purpose.
Three runs against store.example.com tell the real story:
| Run | Estimated fetches | Actual fetches | Variance |
|---|---|---|---|
| Pilot (1,000-row sample) | 1,160 | 1,241 | +7.0% |
| Full crawl (12,000 rows) | 13,946 | 14,406 | +3.3% |
| Re-crawl (12,000 rows) | 13,946 | 15,880 | +13.9% |
The re-crawl blew out because the retailer added cross-listings between runs and the duplicate rate jumped — exactly the kind of drift a budget pretends won’t happen. The report catches it, and the next budget absorbs it.
Before committing to the full crawl, spend 500 fetches and capture: rows-per-page on pages 1, 10, and 20 (the decay curve); the hard-404 versus soft-404 split; redirect share and where redirects land; duplicate rate broken out by category; and the ratio recomputed after every 100 fetches to watch it stabilize. If the ratio hasn’t settled by fetch 500, you don’t understand the site’s structure yet, and that’s a finding, not a failure.
Wrap-Up: Defend the Fetch Count
The fetch-to-row ratio turns “how many rows do you need” into “how many requests will this take,” which is the only question infrastructure and budgets actually answer. Everything reduces to it: dead links divide your yield, duplicates multiply your detail fetches, politeness caps your daily throughput, and retries tax whichever failure class you were too lazy to separate.
Measure it on 500 fetches. Budget with a sensitivity range, not a point. Report actuals against the estimate every run. Do that, and the next time someone asks what 12,000 rows costs, you’ll hand them 14,250 fetches and the math behind it — and the number will hold up.
Related Articles
Price the Bytes: A Storage Budget for Scraped Data
The fetch is priced; the bytes keep billing you. Model storage, gzip, egress, and retention per scraped page with numbers you can recompute.
TechnicalToken Economics of LLM Extraction: A Cost Worksheet
Measure the real token cost of LLM extraction: tokens per page, per field, and per clean record, with a worksheet you can recompute on your own corpus.
TechnicalAmortize Maintenance Into Your Per-Page Cost
Your per-page price omits the hours spent fixing selectors and updating parsers. Learn to amortize scraper maintenance into a true per-page cost model.