On-Demand Scraping vs Prefetched Data: Serving Trade-offs

On-Demand Scraping vs Prefetched Data: Serving Trade-offs

Every scraping system eventually faces the same fork: do you fetch the page when the user asks for it, or do you harvest pages ahead of time and serve from storage? The answer is not “both are fine.” The two architectures have wildly different latency profiles, cost curves, and failure modes, and picking the wrong one shows up as either a 900 ms API response or a product displaying yesterday’s prices.

This comparison breaks down both models with real numbers, break-even math, and the hybrid pattern most production systems converge on. The short version: live scraping caps your freshness at the cost of tail latency and per-request spend; prefetching caps your latency at storage-read speed at the cost of staleness and idle crawl spend. Freshness requirements per data field — not vibes — should drive the decision.

Anatomy of a Request-Path Scrape: Breaking Down the 300–900 ms Live-Fetch Budget

When a scrape runs inside the user’s request, every millisecond of the network stack lands on their response time. People quote “sub-100 ms scraping” in marketing decks. That number is fiction for anything beyond a warmed HTTP/2 connection to a fast origin.

Here is where the time actually goes: DNS resolution (5–40 ms, worse on cold resolvers), TCP connect (one RTT, 10–80 ms depending on geography), TLS handshake (one to two RTTs), time to first byte (server think time, 50–400 ms for dynamic pages), HTML transfer (size over bandwidth), and parse/extract (5–50 ms for a typical product page). Add a rendering step for JavaScript-heavy pages and you are looking at 2–5 seconds, not milliseconds.

Measure it yourself. Don’t trust my numbers or anyone else’s:

import time
import statistics
import httpx

URL = "https://store.example.com/product/42"
N = 50

def timed_fetch(client: httpx.Client) -> dict:
    phases = {}
    t0 = time.perf_counter()
    resp = client.get(URL)
    t1 = time.perf_counter()

    # httpx exposes per-phase timings in response.elapsed and
    # httpcore's extension dict; fall back to coarse split.
    ext = resp.extensions
    network = ext.get("network_stream", None)
    phases["total_ms"] = (t1 - t0) * 1000

    # Parse cost measured separately
    t2 = time.perf_counter()
    _ = resp.text
    t3 = time.perf_counter()
    phases["parse_ms"] = (t3 - t2) * 1000
    return phases

def pct(samples, p):
    ordered = sorted(samples)
    idx = min(int(len(ordered) * p / 100), len(ordered) - 1)
    return ordered[idx]

with httpx.Client(http2=True, follow_redirects=True) as client:
    runs = [timed_fetch(client) for _ in range(N)]

total = [r["total_ms"] for r in runs]
parse = [r["parse_ms"] for r in runs]
print(f"{'metric':<12}{'p50':>10}{'p95':>10}{'p99':>10}")
for name, samples in [("total", total), ("parse", parse)]:
    print(f"{name:<12}{pct(samples,50):>10.1f}{pct(samples,95):>10.1f}{pct(samples,99):>10.1f}")

Typical output from a run against a mid-sized storefront, with per-phase detail from a tracing wrapper:

Phase	p50	p95	p99
DNS	8 ms	35 ms	60 ms
TCP connect	22 ms	90 ms	140 ms
TLS handshake	31 ms	120 ms	190 ms
TTFB	180 ms	420 ms	700 ms
Download (HTML)	25 ms	80 ms	150 ms
Parse + extract	12 ms	40 ms	75 ms
End to end	278 ms	785 ms	1,315 ms

The dominant term is TTFB, and you cannot fix that with a faster client. The server decides how long it thinks. The p99 is nearly 5x the p50, which is the real killer: if your API’s SLO is p99 under 500 ms, a live fetch in the request path fails that SLO on its own, before your own business logic runs.

If the target sits behind anti-bot checks, add the scraping API hop. A call to POST https://api.finedata.ai/api/v1/scrape with use_antibot: true adds its own overhead on top of the target’s TTFB — cheaper than getting blocked, but not free in latency terms. For a deeper treatment of why the TLS handshake itself is a detection vector, see TLS Fingerprinting Explained: How Anti-Bot Systems Detect Scrapers.

The Prefetch Pipeline: Scheduled Harvesting and the Freshness Window You Accept

The prefetch model inverts the flow. You crawl on a schedule, normalize the results, load them into Postgres or Redis or S3-plus-Parquet, and your API reads from storage. User-facing latency becomes a primary-key lookup: 1–5 ms. The price is staleness, and staleness is a number you can compute exactly.

A scheduled harvester config looks something like this:

# harvester.yaml
schedule: "*/30 * * * *"        # every 30 minutes
concurrency: 20
timeout_seconds: 60
max_retries: 3
targets:
  - source: "https://example.com/sitemap.xml"
    type: sitemap
    filter: "^https://store.example.com/product/"
  - source: "https://store.example.com/collections/sale"
    type: listing
    extract_rules:
      product_url: "a.product-card@href"
      price: ".price-now"
storage:
  kind: postgres
  table: product_snapshots
  dedupe_key: [url, fetched_at]
notifications:
  on_failure_rate_above: 0.10

The freshness math is unforgiving and worth writing down. Worst-case staleness for any record is not the crawl interval — it is the interval plus queue lag plus load time:

crawl_interval = 30 * 60      # seconds, from the schedule
queue_lag      = 4 * 60       # seconds, jobs waiting for a worker slot
load_time      = 90           # seconds, normalize + bulk insert

worst_case_staleness = crawl_interval + queue_lag + load_time
print(f"{worst_case_staleness / 60:.1f} minutes")   # 34.5 minutes

So a “30-minute freshness” pipeline actually serves data up to ~35 minutes old, and on average about half the interval — around 17 minutes. If your product requirement is “price shown within 5 minutes of change,” a 30-minute schedule does not meet it, full stop. Tighten the schedule and your crawl spend rises linearly while freshness improves linearly. There is no clever trick here; the trade is linear both ways, which is at least honest.

One underappreciated benefit: prefetch failures happen offline. A 3% failure rate during a 3 AM crawl is a log line and a retry. The same failure rate in the request path is a user-visible error. More on that below.

Cost per Served Result: Break-Even Math Between Live Scraping and Storage

Latency is only half the decision. The other half is cost per served result, and this is where read volume dominates.

Cost line	On-demand (live scrape)	Prefetched (storage)
Requests to target (or API)	1 per user request	1 per crawl cycle, regardless of reads
Compute	Scraping worker per request	Crawl workers on schedule only
Bandwidth/proxy	Per request, scales with traffic	Fixed by crawl volume
Storage	~0	Object store + database, grows with corpus
Cost per 1M served results	1M fetches	1 crawl cycle worth of fetches + storage
Latency p50	250–900 ms	1–5 ms
Freshness	Real-time	Up to crawl interval + lag

The break-even point is simple arithmetic. Assume a scraping API charges roughly per successful fetch, and your crawl covers 50,000 URLs on a 30-minute schedule — 48 cycles a day, so 2.4M fetches per day just to keep the corpus warm. If users read each record 10 times a day, on-demand serving costs 500,000 fetches per day. Prefetching costs nearly 5x more in fetch spend, plus storage, and delivers data that is up to half an hour old.

Run it the other way:

corpus_size = 50_000
crawl_interval_min = 30
cycles_per_day = 24 * 60 / crawl_interval_min       # 48
reads_per_record_per_day = 2                        # low-traffic product

prefetch_fetches = corpus_size * cycles_per_day     # 2.4M/day
live_fetches = corpus_size * reads_per_record_per_day  # 100k/day

# Live is cheaper while: reads_per_day < cycles_per_day
break_even_reads = cycles_per_day
print(f"Break-even: {break_even_reads:.0f} reads/record/day")
# Below 48 reads per record per day, live scraping is cheaper.
# Above it, prefetching wins on fetch spend alone.

The general rule falls out of the formula: prefetching is cheaper when reads per record per day exceed crawl cycles per day. A price-monitoring dashboard with 5 daily views per product and an hourly crawl is a prefetch case. A long-tail product lookup where 99% of records get read once a month is emphatically a live-fetch case — prefetching that corpus is paying to refresh data nobody looks at. For more on this trade at the billing-model level, Success-Based vs Metered Scraping API Billing Models covers how metered pricing shifts the math.

What a Failed Fetch Actually Costs: Timeouts, Retries, and Tail Latency Amplification

Here is where live scraping gets genuinely expensive, and where most back-of-napkin cost comparisons go wrong. They model the happy path. The happy path is not the problem.

Say 3% of live fetches fail — target rate-limits you, a proxy times out, the page 500s. Naive instinct is “3% more cost, 3% error rate.” Reality is worse on both axes, because retries multiply latency at exactly the tail you care about:

import random, time

def fetch_with_backoff(url: str, max_retries: int = 4, base: float = 0.5):
    for attempt in range(max_retries + 1):
        try:
            resp = scrape(url)   # your fetch layer
            if resp.ok:
                return resp
            if resp.status_code in (429, 500, 502, 503):
                raise TransientError(resp.status_code)
            raise FatalError(resp.status_code)   # 403, 404: don't retry
        except TransientError:
            if attempt == max_retries:
                raise
            sleep = base * (2 ** attempt) + random.uniform(0, 0.25)
            time.sleep(sleep)

Model a 3% transient failure rate with 4 retries and exponential backoff starting at 500 ms:

Metric	Value
Requests that need 1 retry	3.0%
Requests that need 2+ retries	~0.09%
Added p95 latency (one backoff cycle)	+500–1,000 ms
Added p99 latency (two cycles)	+1,500–3,000 ms
Wasted fetch spend	~3.1% of traffic
User-visible error rate (all retries fail)	~0.0001% if failures are independent, ~1–3% if correlated

That last row is the trap. Failures are rarely independent. When a target rate-limits you, it rate-limits a burst of your traffic at once, and your retry storm hammers a server that is already refusing you. Correlated failures turn “3% retry overhead” into “3% of users see a 10-second timeout followed by an error.” I have seen teams set max_retries: 5 with no jitter and effectively DDoS their own scraping quota within seconds of a target hiccup.

Prefetch pipelines absorb all of this invisibly. A failed fetch at 3 AM retries during the next cycle, or the next, and the user never knows — they just see data that is one interval staler. The failure cost moves from your p99 and your error budget into a background metric. That asymmetry, more than raw latency, is the strongest architectural argument for prefetching anything with read volume.

Freshness as the Decision Variable: Mapping Data Types to TTL and Stale-While-Revalidate

The framing that finally makes this decision tractable: stop choosing an architecture per site. Choose per field. A product record is not uniformly fresh — its price changes hourly, its description changes yearly.

Map each field to a freshness requirement, then assign a strategy:

Field	Volatility	Acceptable staleness	Strategy
Price	High	Minutes	Short TTL, live on miss
Stock status	High	Minutes	Short TTL
Title	Low	Hours–days	Long TTL
Description	Very low	Days	Prefetch only
Images	Near-zero	Weeks	Prefetch, cache forever

In Redis or an app-level cache, that becomes tiered TTLs:

import json, redis, time

r = redis.Redis()

FIELD_TTL = {
    "price": 60,           # 1 minute
    "stock": 60,
    "title": 6 * 3600,
    "description": 24 * 3600,
    "images": 7 * 24 * 3600,
}

def get_product(url: str):
    pipe = r.pipeline()
    for field, ttl in FIELD_TTL.items():
        pipe.get(f"p:{url}:{field}")
    values = pipe.execute()
    record = {}
    stale_fields = []
    for (field, ttl), raw in zip(FIELD_TTL.items(), values):
        if raw is None:
            stale_fields.append(field)
        else:
            record[field] = json.loads(raw)
    if stale_fields:
        enqueue_refresh(url, stale_fields)   # background job
    return record, stale_fields

Then layer stale-while-revalidate at the serving edge so a stale hit costs nothing in latency. In nginx:

proxy_cache_path /var/cache/nginx keys_zone=pages:50m;

server {
    location /product/ {
        proxy_cache pages;
        proxy_cache_valid 200 10m;
        proxy_cache_use_stale error timeout updating
                              http_500 http_502 http_503 http_504;
        proxy_cache_background_update on;
        proxy_pass https://store.example.com;
    }
}

With proxy_cache_use_stale updating plus background update, a request during refresh gets the old copy instantly while the new one loads. The user sees a price up to one TTL old; whether that is acceptable is a product decision, not an engineering one. My opinion, which you may disagree with: for most e-commerce display use cases, a 60-second-old price is fine, and teams that demand real-time prices for a browse UI are usually importing a checkout-page requirement into a catalog page. Reserve live-in-path scraping for the moments that actually transact.

The Hybrid Pattern: Serve Stored First, Scrape on Miss, Refresh in the Background

Production systems almost never run pure on-demand or pure prefetch. They run cache-aside with background refresh, which caps user latency at storage-read speed while still guaranteeing eventual freshness and absorbing failures offline:

import time, json, redis
import httpx

r = redis.Redis()
FRESH_TTL = 60          # seconds a record is considered fresh
HARD_MISS_TIMEOUT = 8.0 # seconds we'll make a user wait on a cold record

def serve_product(url: str):
    # Path 1: fresh hit — serve from storage, no scrape
    raw = r.get(f"product:{url}")
    if raw:
        record = json.loads(raw)
        if time.time() - record["fetched_at"] < FRESH_TTL:
            return record, "hit"

    # Path 2: stale hit — serve old data, refresh in background
    if raw:
        enqueue_refresh(url)          # async worker scrapes and rewrites
        return json.loads(raw), "stale-hit"

    # Path 3: hard miss — scrape inline, bounded by a timeout
    try:
        record = live_scrape(url, timeout=HARD_MISS_TIMEOUT)
        r.set(f"product:{url}", json.dumps(record))
        return record, "miss-inline"
    except (httpx.TimeoutException, httpx.HTTPError):
        # Path 4: miss and failed fetch — degrade, don't crash
        return {"url": url, "available": False}, "miss-error"

def live_scrape(url: str, timeout: float) -> dict:
    resp = httpx.post(
        "https://api.finedata.ai/api/v1/scrape",
        headers={"Authorization": "Bearer fd_your_api_key"},
        json={
            "url": url,
            "formats": ["text"],
            "extract_rules": {
                "title": "h1",
                "price": ".price-now",
                "stock": ".availability",
            },
            "timeout": 60,
        },
        timeout=timeout,
    )
    resp.raise_for_status()
    data = resp.json()
    return {"fetched_at": time.time(), **data.get("data", {})}

The four request paths, and what each costs:

Path	Condition	User latency	Scrape cost	Failure exposure
Cache hit	Record fresh	1–5 ms	None	None
Stale hit + refresh	Record older than TTL	1–5 ms	1 background fetch	None (user served)
Miss + inline scrape	No record stored	300–900 ms	1 fetch	Timeout, error page
Miss + fallback	Inline scrape failed	Fast error or degraded payload	1 wasted fetch	Handled

The elegance is in path 2. The refresh happens after the response is sent, so scrape latency never touches the user, and a failed refresh just leaves the stale record in place for the next cycle to fix. Path 3 is the only one with real user-facing risk, and it only happens once per record — after that, the record exists in storage and paths 1 and 2 take over.

If your refresh workers need to survive bursts, submit them as async jobs rather than blocking a worker thread — the batch and webhook flow is covered in Async Scraping at Scale: Jobs, Batches, Webhooks.

Running Your Own Break-Even Benchmark: A Local Measurement Harness

Every number in this post is a placeholder for your numbers. Your targets, your geography, your failure rates. Here is a harness that measures the three quantities that matter — live latency, prefetched latency, and failure rate — so the decision is empirical:

import time, statistics, httpx, json

TARGETS = [
    "https://example.com",
    "https://store.example.com/product/42",
    "https://store.example.com/pricing",
]
RUNS = 30

def bench_live(client, url):
    t0 = time.perf_counter()
    try:
        resp = client.get(url, timeout=10)
        ok = resp.status_code == 200
    except httpx.HTTPError:
        ok = False
    return (time.perf_counter() - t0) * 1000, ok

def bench_prefetched(url):
    t0 = time.perf_counter()
    with open(f"cache/{url.replace('/', '_')}.json") as f:
        json.load(f)
    return (time.perf_counter() - t0) * 1000, True

results = {}
with httpx.Client(http2=True) as client:
    for url in TARGETS:
        live = [bench_live(client, url) for _ in range(RUNS)]
        cached = [bench_prefetched(url) for _ in range(RUNS)]
        lats = [t for t, _ in live]
        fails = sum(1 for _, ok in live if not ok)
        results[url] = {
            "live_p50_ms": statistics.median(lats),
            "live_p95_ms": sorted(lats)[int(RUNS * 0.95) - 1],
            "live_fail_pct": 100 * fails / RUNS,
            "cached_p50_ms": statistics.median([t for t, _ in cached]),
        }

for url, m in results.items():
    print(f"{url}: live p50={m['live_p50_ms']:.0f}ms "
          f"p95={m['live_p95_ms']:.0f}ms fail={m['live_fail_pct']:.0f}% "
          f"cached p50={m['cached_p50_ms']:.1f}ms")

Sample output from a local run:

URL	Live p50	Live p95	Fail rate	Cached p50
example.com	142 ms	310 ms	0%	0.4 ms
store.example.com/product/42	340 ms	810 ms	3%	0.4 ms
store.example.com/pricing	410 ms	1,150 ms	7%	0.4 ms

Then apply the checklist to each URL class:

1. Reads per record per day vs crawl cycles per day — if reads exceed cycles, prefetch. This is the break-even test from earlier.
2. Measured failure rate — anything above ~2% in the request path needs the hybrid pattern with inline-scrape timeouts, not naive live serving.
3. Freshness requirement per field — if the only volatile field is price, tier TTLs instead of scraping the whole page live.
4. Tail latency budget — if your API SLO is p99 under 500 ms, live scraping of anything slower than ~300 ms p50 is off the table.
5. Corpus churn — long-tail URLs that are rarely read should never be prefetched; they fail the break-even test by definition.

Wrap-up

On-demand scraping and prefetched storage are not competitors; they are endpoints of a freshness-versus-latency spectrum, and the hybrid cache-aside pattern occupies the sensible middle. Live fetches buy real-time data at the cost of a 300–900 ms p50, a fat p99, and per-read spend. Prefetching buys millisecond serving at the cost of a freshness window you can compute to the minute and crawl spend that scales with corpus size, not traffic.

Three rules I would commit to before writing any code: decide per field, not per site — price and description do not deserve the same strategy. Compute the break-even (reads per record per day vs crawl cycles per day) before choosing prefetching, because prefetching a long-tail corpus is pure waste. And never let an unbounded retry loop run in the request path — correlated failures will find it.

Start with the hybrid pattern, measure with your own harness, and let the numbers move individual URL classes between live and prefetched over time. The architecture that serves from storage first, scrapes on miss, and refreshes in the background is the one that survives contact with real traffic.