Parse at Fetch Time vs Store Raw HTML: Pipeline Trade-offs

Two Pipeline Shapes: Where the Parser Sits Determines What a Failure Costs

Every scraping pipeline answers one deceptively simple question: when do you parse? You can extract structured data the moment a page lands, or you can archive the raw HTML and parse later — possibly much later, possibly never. That single ordering decision ripples through everything: your storage bill, your recovery story after a parser bug, your exposure to schema drift, and how much a failed request actually costs you.

Here are the two shapes:

Pipeline A: parse-at-fetch
  fetch -> parse -> validate -> store parsed JSON
                    |
                    +-- fetch fails? retry fetch.
                    +-- parse fails? re-fetch the page. (ouch)

Pipeline B: store-raw-first
  fetch -> store raw HTML -> parse-on-demand -> validate -> store parsed JSON
                                        |
                                        +-- parse fails? re-parse the blob. No network.
                                        +-- new schema? replay the archive.

The composition makes the difference concrete:

# Pipeline A: parse-at-fetch
def pipeline_a(url):
    html = fetch(url)            # network call, the expensive part
    record = parse(html)         # if this throws, html is gone
    save_json(record)            # raw bytes discarded

# Pipeline B: store-raw-first
def pipeline_b(url):
    html = fetch(url)
    save_raw_blob(html)          # archive first
    record = parse(html)          # failure here is cheap to redo
    save_json(record)

Two lines of difference. Radically different failure economics.

In Pipeline A, the parse stage holds the raw bytes only in memory. Any bug in your selector logic, any unexpected template variant, and the only way to reproduce the input is to go back to the network. If the site has changed since the crawl — and it probably has — you can’t even reproduce the bug faithfully. The blast radius of a parser defect is a re-crawl.

In Pipeline B, the archive is the source of truth. Parsed JSON becomes a derived, disposable view. Bugs get fixed by replaying, not re-fetching. The trade-off is that you pay for every byte twice: once in storage, once in the discipline required to keep the archive from becoming an unmanaged dump.

Neither shape is universally correct. The rest of this post is about when each one wins.

Reprocessing After a Parser Bug: Re-running Extraction Over 10M Stored Pages Without Re-fetching

The scenario that decides the argument: you crawled 10M product pages, built a dataset on top, and three weeks later someone finds that your price parser silently dropped the “was” price on every page that had a discount banner. Pipeline A says: re-crawl. Pipeline B says: point the new parser at the archive.

import gzip
import json
from pathlib import Path

ARCHIVE = Path("/mnt/store.example.com/blobs")

def parse_product_v1(html: str) -> dict:
    # Old logic: assumed a single price node
    return {"price": first_match(html, "span.price")}

def parse_product_v2(html: str) -> dict:
    # Fixed: handles discount layout, normalizes currency
    price = first_match(html, "span.price") or first_match(html, "span.was-price")
    return {
        "price": price,
        "discounted": bool(first_match(html, "span.was-price")),
    }

def replay_archive(parser):
    fixed = 0
    for blob_path in ARCHIVE.glob("*.html.gz"):
        with gzip.open(blob_path, "rt", encoding="utf-8") as f:
            html = f.read()
        record = parser(html)
        if record["price"] is not None:
            write_parsed(blob_path.stem, record)  # overwrite derived record
            fixed += 1
    return fixed

replay_archive(parse_product_v2)

The replay is a pure CPU job. It parallelizes trivially, needs no politeness delays, and — critically — reproduces the exact input that produced the bug, not a newer version of the page that may have changed since.

Compare the recovery cost for a 10M-page corpus:

Recovery path	Wall-clock time	Bandwidth	Politeness cost	Site-dependency risk
Re-crawl example.com (10M pages, ~2 req/s sustained)	~58 days single-threaded; ~1 day at 120 req/s across workers	~4 TB (10M × ~400 KB)	Full crawl etiquette, rate limits, possible blocks	Pages may have changed; bug may not reproduce
Re-parse stored HTML (10M blobs, ~20ms parse each)	~56 CPU-hours; under 1 hour on a 64-core box	~0	None	None — exact original input

That bandwidth column deserves attention. Re-crawling 10M pages is not free even if you use a scraping API; at roughly 400 KB per page you’re paying for 4 TB of transfer and 10M successful fetches. The re-parse costs electricity. If your extraction logic changes more than once a quarter — and for anything scraping templated sites, it will — the archive pays for itself fast.

This is also the strongest argument against the “just re-scrape it” reflex I hear from teams running lean pipelines: re-fetching is not a retry, it’s a second data acquisition with its own failure modes, its own cost, and — worst of all — different data than what you originally parsed.

Schema Drift on example.com: Catching Selector Breakage Before It Corrupts Your Dataset

Target sites redesign. When example.com ships a new template, your span.price selector starts returning nothing. How each pipeline experiences this is very different.

Parse-at-fetch pipelines tend to fail loudly, which is actually a feature: null fields flow into your validation layer immediately, and if you validate, you catch the drift the same day it happens. The risk is silent partial breakage — the selector still matches something, just the wrong node, and you write garbage prices for a week before anyone notices.

Store-raw-first pipelines have the opposite failure profile: the archive keeps filling with healthy HTML regardless of parser state, so drift never corrupts the source. But your derived dataset can drift silently for as long as nobody looks at it, because nothing in the fetch path depends on the parser working.

Either way, you need validation at the parse boundary. Pydantic works well here:

from pydantic import BaseModel, field_validator

class ProductRecord(BaseModel):
    url: str
    title: str
    price: float

    @field_validator("price")
    @classmethod
    def price_sane(cls, v: float) -> float:
        if v <= 0 or v > 1_000_000:
            raise ValueError(f"implausible price: {v}")
        return v

# In the parse loop:
try:
    ProductRecord(**raw_extract)
except ValidationError as e:
    alert(f"schema drift on example.com: {e}")
    quarantine(raw_extract)  # don't let it reach the dataset

Validation catches the record, not the trend. For trend detection, track per-field null rates:

-- Daily null-rate per field, computed from parsed records
SELECT
    DATE(parsed_at) AS day,
    AVG(CASE WHEN price IS NULL THEN 1 ELSE 0 END) AS price_null_rate,
    AVG(CASE WHEN title IS NULL THEN 1 ELSE 0 END) AS title_null_rate
FROM parsed_products
WHERE source = 'example.com'
GROUP BY DATE(parsed_at)
ORDER BY day DESC;

day        | price_null_rate | title_null_rate
2026-01-11 | 0.002           | 0.001     <- baseline noise
2026-01-12 | 0.417           | 0.003     <- template changed; ALERT

Alert when a field’s null rate exceeds, say, 5% over its trailing 7-day baseline. A single null record is noise; a null rate that jumps an order of magnitude overnight is a redesign, and you want to know about it before your downstream analytics bake the gap into a quarterly number.

If you run raw-first, drift detection has a bonus property: because the healthy HTML is already archived, fixing the selector and replaying just the affected date range is a batch job, not an incident. If you run parse-at-fetch, the drift window is a hole in your dataset that no amount of cleverness will fill — the pages are gone. That asymmetry is worth more than any storage optimization.

Storage Math: Raw HTML Bytes vs Parsed JSON at Corpus Scale

The honest cost of raw-first. Measure it yourself before projecting:

# Raw page size
curl -s https://example.com | wc -c
# e.g. 41200 bytes

# Parsed record size (one row of your JSON output)
echo '{"url":"https://example.com/p/1","title":"Widget","price":19.99}' | wc -c
# e.g. 74 bytes

# What a stored sample actually costs on disk
du -sh /mnt/store.example.com/samples/

Typical ratios: a full HTML page runs 100 KB to 1 MB; the parsed record is usually 100 bytes to 2 KB. That’s a 100x to 500x size difference per page. Project it:

Corpus size	Raw HTML (400 KB avg)	zstd-compressed HTML (~12% of raw)	Parsed JSON (1 KB avg)
1M pages	400 GB	48 GB	1 GB
10M pages	4 TB	480 GB	10 GB
100M pages	40 TB	4.8 TB	100 GB

Two observations from that table. First, uncompressed raw storage is genuinely brutal at 100M pages — 40 TB is real money on object storage even at commodity prices. Second, compression changes the picture completely: HTML is extremely compressible (repeated boilerplate, markup tags), and zstd routinely gets 8–12x on web pages. Compressed raw at 10M pages is under half a terabyte — a rounding error in most infra budgets.

My rule of thumb: if compressed raw storage costs less than 10% of what a single full re-crawl of the same corpus costs, archive it. That threshold clears easily for anything you scrape more than once.

Retry Economics: What a Failed Fetch and a Failed Parse Each Cost You

Retries are where the two pipelines waste different resources. Fetch failures waste bandwidth and politeness budget; parse failures waste CPU and, in the parse-at-fetch shape, force a re-fetch of a page you already had in hand.

Failure type	Parse-at-fetch: wasted resource	Raw-first: wasted resource	Retry cost
Network error / timeout	Bandwidth, request budget	Same — but blob simply never written	Cheap: retry fetch, no parse work done
Anti-bot block (4xx/403)	Bandwidth, proxy/API cost	Same	Expensive either way; needs different exit IP or stealth settings
Parse exception	Full fetch wasted — HTML discarded on error	CPU only; blob retained, re-parse locally	Raw-first converts a re-fetch into a re-compute
Partial extraction (null fields)	Often unnoticed; pollutes dataset	Blob retained; can re-parse after fixing	Raw-first: near-zero. Parse-at-fetch: silent data loss
Validation failure	Full fetch wasted if you discard	Blob retained; quarantine record, fix parser, replay	Same as parse exception

The per-stage retry split matters. Fetch retries should be aggressive with backoff, because transient network failures are common and retrying is the correct response. Parse retries should barely exist — a parser that fails twice on the same input will fail a third time; log it and quarantine instead of burning CPU in a loop.

# Fetch stage: generous retries, backoff, service-side retry offload
FETCH_RETRY = {
    "max_retries": 5,        # let the API retry internally
    "backoff_base": 2.0,
    "retry_statuses": [429, 500, 502, 503, 504],
    "timeout": 120,
}

# Parse stage: no blind retries — quarantine and move on
PARSE_RETRY = {
    "max_retries": 1,
    "on_failure": "quarantine",   # keep the blob, flag the record
    "alert_threshold": 0.05,     # 5% parse failure rate pages a human
}

If you offload fetching to a scraping API, the same split applies: set max_retries and timeout generously on the fetch side and let the service absorb transient failures, but never feed a parse failure back into a re-fetch. The API can retry a request it made; it cannot fix a selector you wrote. (Client-side versus service-side retry budgets are worth a read if this split is new to you — see client-side vs service-side retries.)

One more subtlety: partial extractions are the failure mode nobody budgets for. A page that returns a title but a null price looks successful to a naive pipeline. Raw-first pipelines survive partial extraction because the blob is still there when you fix the field-level bug. Parse-at-fetch pipelines don’t — the null becomes permanent.

Taming the Raw Store: Compression, Content-Addressed Dedup, and Retention Tiers

The raw archive only stays viable if you manage it. Three techniques do most of the work.

Compression. Always. zstd level 3 gives near-gzip ratios at several times the speed, and speed matters when you’re writing millions of blobs.

Content-addressed storage. Key blobs by a hash of normalized content rather than by URL. Identical pages deduplicate automatically, and re-crawls of unchanged pages cost you nothing in storage. On a product corpus where 30–40% of pages are unchanged between daily crawls, that’s a third of your storage gone before retention policy even kicks in.

import hashlib
import zstandard as zstd
from pathlib import Path

ARCHIVE = Path("/mnt/store.example.com/blobs")
ARCHIVE.mkdir(parents=True, exist_ok=True)

def normalize(html: str) -> str:
    # Strip volatile bits before hashing, or dedup misses near-identical pages
    return "".join(html.split()).lower()

def store_blob(html: str) -> str:
    key = hashlib.sha256(normalize(html).encode()).hexdigest()[:32]
    blob_path = ARCHIVE / f"{key}.zst"
    if blob_path.exists():
        return key  # dedup hit — already archived
    compressor = zstd.ZstdCompressor(level=3)
    blob_path.write_bytes(compressor.compress(html.encode("utf-8")))
    return key

Retention tiers. Not everything deserves the same retention. A hot tier (30 days, fast storage) covers the window where parser bugs and drift get discovered — realistically, most replay jobs run against the last few weeks of data. A cold tier (compressed, object storage, optional) covers archival and audit needs. Anything older than your fix-discovery window with no legal or research requirement can be dropped outright.

Measured on a 100k-page crawl of a templated site, with a 30-day hot / cold split:

Storage variant	Size	Notes
Raw HTML, uncompressed	40 GB	100k pages × ~400 KB
gzip (-6)	5.1 GB	~8x reduction, slower
zstd (-3)	4.6 GB	~8.7x reduction, much faster write/read
zstd + content-addressed dedup (daily re-crawl, ~35% unchanged)	3.0 GB effective	Dedup compounds across crawl cycles
After 30-day hot-tier expiry (drop cold tier)	~0.4 GB steady-state	Only last 30 days retained hot

The steady-state number is the one to internalize: with dedup and a sane retention window, an ongoing daily crawl of 100k pages costs well under a gigabyte of hot storage. The “raw storage is expensive” objection mostly evaporates under actual management — it only holds for naive uncompressed, dedup-free, infinite-retention archives, which nobody should run.

A Per-Source Decision Framework: Mixing Both Modes Instead of Picking One

The framing of “parse-at-fetch vs raw-first” as a global choice is wrong. Real pipelines handle dozens of sources with wildly different volatility, volume, and value. The correct architecture runs both modes and assigns them per source.

# pipeline_modes.yaml
sources:
  example.com:
    mode: parse_at_fetch
    rationale: stable template, low per-page value, high volume
    validate: strict
    archive: none

  store.example.com:
    mode: store_raw_first
    rationale: volatile pricing pages, high downstream value
    archive:
      compression: zstd
      dedup: content_addressed
      hot_retention_days: 30
      cold_retention_days: 365
    validate: strict

The rubric I use:

Criterion	Favors parse-at-fetch	Favors store-raw-first
Page volume	Very high, low value per page	Moderate volume, high value per page
Template churn	Stable for months	Redesigns or A/B tests frequently
Extraction maturity	Parser battle-tested, schema stable	New or brittle selectors, schema still evolving
Downstream dependency	Data consumed once, ephemeral	Dataset feeds analytics; historical integrity matters
Legal / archival need	None	Compliance audit, research reproducibility
Re-parse likelihood	Low	High — extraction logic still changing

Two judgment calls worth defending. First: parse-at-fetch is underrated for stable, boring sources. If a template hasn’t changed in six months and your parsed record is 1 KB against a 400 KB page, archiving the raw HTML is paying a 400x storage tax to insure against a risk that has already stopped materializing. Insure what’s volatile; stop insuring what isn’t.

Second, and more contentious: for genuinely high-value sources, I’d store raw even when the parsed data looks perfect. The reason is that you don’t know today what you’ll want to extract next year. The product pages you scrape for prices today contain review counts, shipping terms, and stock indicators that a future requirement will suddenly need — and with a raw archive, that new requirement is a replay job, not an archaeological expedition into pages that no longer exist. Historical HTML is unobtainable after the fact; storage is merely billable. That asymmetry is the whole argument.

If you’re weighing this against the other classic build-vs-buy question — whether to manage fetch infrastructure at all — the scraping API vs DIY cost analysis covers the acquisition side, and building ETL pipelines with scraping APIs covers how raw archives slot into a broader ETL shape.

Wrap-up

Where the parser sits is an insurance decision. Parse-at-fetch is cheap on storage and fast to ship, but every parser bug, template change, and new extraction requirement becomes a re-crawl — and a re-crawl is a second data acquisition, not a retry. Store-raw-first costs compression and discipline, and buys you replayability, drift immunity for the source of truth, and the ability to answer extraction questions you haven’t thought of yet.

The practical answer is per-source assignment: parse-at-fetch for stable high-volume low-value sources, raw-first with dedup and tiered retention for anything volatile or valuable. Measure your own ratios — page bytes to record bytes, churn rate, re-parse frequency — and let the numbers pick the mode. The only genuinely wrong answer is running one mode globally because the architecture diagram was easier to draw that way.