Price the Bytes: A Storage Budget for Scraped Data

Why the Fetch Is the Cheap Part

Storage is the line item nobody models. We price the fetch down to the token, we argue about retry costs, and then we let the raw HTML pile up in a bucket with a shrug — “storage is cheap.” It is cheap per gigabyte. It is not cheap once you multiply it by pages per day, retention, request meters, and the egress your reprocessing pipeline quietly generates every night.

This post builds the missing budget. Every number here is recomputable: you sample your own pages, measure your own compression ratios, and plug in current list prices from your provider. No hand-waving.

Measure First: Building a Per-Page Byte Profile from example.com

Every storage model starts with one number: average bytes per page. Not the number you guessed. The number you measured.

The sampler below submits 200 URLs as an async batch, waits for completion, and logs three sizes per page: the raw HTML as returned, the decompressed byte count, and the size after stripping <script> and <style> blocks. The stripped number matters because it tells you how much of the page is actually content versus boilerplate you will never parse — inline JS bundles alone can be half the payload on modern storefronts.

import csv, json, re, time, requests

API = "https://api.finedata.ai"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}

urls = [f"https://example.com/items/{i}" for i in range(200)]

payload = {
    "requests": [
        {"url": u, "formats": ["rawHtml"], "only_main_content": False}
        for u in urls
    ]
}

r = requests.post(f"{API}/api/v1/async/batch", headers=HEADERS, json=payload)
batch_id = r.json()["batch_id"]

# Poll until the batch finishes
while True:
    s = requests.get(
        f"{API}/api/v1/async/batch/{batch_id}",
        headers=HEADERS,
        params={"include_results": True},
    ).json()
    if s.get("status") in ("completed", "failed", "cancelled"):
        break
    time.sleep(10)

strip_re = re.compile(r"<(script|style)[^>]*>.*?</\1>", re.S | re.I)

with open("page_sizes.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["url", "raw_bytes", "stripped_bytes"])
    for job in s.get("jobs", []):
        html = job.get("result", {}).get("rawHtml")
        if not html:
            continue
        raw = html.encode("utf-8")
        stripped = strip_re.sub("", html).encode("utf-8")
        w.writerow([job.get("url"), len(raw), len(stripped)])

Run this against your actual target domain, not a placeholder. Two hundred pages is enough for a stable median on a homogeneous site; use a thousand if the corpus mixes product pages, category pages, and articles, because those have wildly different size distributions.

Here is what a typical run looks like on a content-heavy, JS-bloated site:

Metric	Raw HTML	Stripped (no scripts/styles)
Median	190 KB	84 KB
p90	480 KB	205 KB
Mean	238 KB	96 KB

Two things jump out. First, the p90 is 2.5× the median — a few enormous pages drag the mean up, so budget on the mean, not the median, or you will undersize the bucket. Second, stripping scripts and styles removes 60% of the bytes. That is the single biggest storage decision you will make, and it is a policy question, not a compression question: do you store the raw snapshot or the cleaned content? I store raw, because extraction logic breaks and you will want to re-parse the original bytes — but that opinion costs you 2.5× the storage, and you should make it deliberately. The trade-offs are laid out in Parse at Fetch Time vs Store Raw HTML.

Baseline for the rest of this post: 238 KB mean raw page, 55 KB after gzip -6 (measured in the next section).

The Storage Equation: Pages per Day × Bytes × Retention Days

The core formula is three multiplications:

standing_volume = pages_per_day × avg_page_bytes × retention_days

At steady state, your bucket holds exactly the last retention_days of intake. During the ramp-up you hold less; after the ramp, the oldest pages expire as fast as new ones arrive. The billed quantity is GB-months, which at steady state equals the standing volume.

def accrued(pages_per_day: int, avg_page_bytes: int, retention_days: int):
    gb = pages_per_day * (avg_page_bytes / 1024**3) * retention_days
    gb_months = pages_per_day * (avg_page_bytes / 1024**3) * 30
    return {"standing_gb": round(gb, 1), "monthly_intake_gb_months": round(gb_months, 1)}

Plugging in the baseline (55 KB compressed per page, 90-day retention):

Pages per day	Daily intake	Standing volume at 90 days
10,000	0.55 GB	49.5 GB
100,000	5.5 GB	495 GB
1,000,000	55 GB	4,950 GB (4.95 TB)

Notice what the equation says: retention and fetch rate are symmetric multipliers. Doubling retention costs exactly as much as doubling your crawl rate. A 365-day retention window on a 100K-per-day pipeline is 2 TB of standing HTML — and, as you will see below, 2 TB that your reprocessing jobs will keep pulling back out and paying for again.

The number that should worry you is not the TB. It is that at 1M pages per day, you are issuing a billion PUTs per year. Hold that thought for the pricing section.

Compression Reality Check: gzip and Brotli Ratios on Real Scraped HTML

Everyone assumes “HTML compresses about 10×.” It does not. Scraped HTML is full of high-entropy inline JSON, base64 images, and minified JS — compressible, but nowhere near text-book ratios. Measure it on your own corpus:

import gzip, brotli, statistics
from pathlib import Path

corpus = [p.read_bytes() for p in Path("snapshots").glob("*.html")]

def ratio(data: bytes, compressed: bytes) -> float:
    return len(data) / len(compressed)

raw = sum(len(d) for d in corpus)
g6  = sum(len(gzip.compress(d, 6))  for d in corpus)
g9  = sum(len(gzip.compress(d, 9))  for d in corpus)
b5  = sum(len(brotli.compress(d, quality=5))  for d in corpus)
b11 = sum(len(brotli.compress(d, quality=11)) for d in corpus)

for name, size in [("gzip -6", g6), ("gzip -9", g9),
                   ("brotli q5", b5), ("brotli q11", b11)]:
    print(f"{name:12} {size/raw:.2f}:1   saved {100*(1-size/raw):.1f}%")

Typical output on the sampled corpus:

Codec	Ratio	Bytes saved
gzip -6	4.3:1	76.5%
gzip -9	4.4:1	77.0%
Brotli q5	4.6:1	78.3%
Brotli q11	5.1:1	80.4%

The spread between gzip -6 and Brotli q11 is about 4 percentage points of the original size. Brotli q11 gets there by being brutally slow on the write path — often 10-20× the CPU time of gzip -6. For write-once, read-rarely snapshot storage, that CPU spend is not worth it. My recommendation: gzip -6 as the default, Brotli q5 only if you are already CPU-rich on the write path. gzip -9 is the worst of both worlds — meaningful extra CPU for a rounding-error gain.

One more thing the table does not show: some object stores decompress on GET when you set Content-Encoding: gzip, which silently converts your storage savings into egress volume. Compress the payload yourself and upload opaque compressed bytes unless you specifically want transparent decompression.

import gzip, requests

def upload_snapshot(html: str, key: str):
    body = gzip.compress(html.encode("utf-8"), 6)
    return requests.put(
        f"https://store.example.com/{key}.html.gz",
        data=body,
        headers={"Content-Type": "application/gzip"},
    )

Line-Item Pricing: Object Storage Rates Applied to the Baseline Volume

Here are the standard-tier list prices for the three major providers, single-region, as of this writing. Verify current rates before you budget — these move.

Line item	AWS S3 Standard	GCS Standard	Azure Blob Hot (LRS)
Storage ($/GB-month)	$0.023	$0.020	$0.018
Write / PUT ($/1K requests)	$0.005	$0.005	$0.0065
Read / GET ($/1K requests)	$0.0004	$0.0004	$0.0052
Internet egress ($/GB)	$0.09	$0.12	$0.087
Minimum billable object size	none	none	none (but min 30-day storage on some tiers)

Now the worked arithmetic for the 100K-pages-per-day baseline, 55 KB compressed, stored in store.example.com on S3 Standard, 90-day retention:

• Standing volume: 100,000 × 55 KB × 90 = 495 GB
• Storage: 495 GB × $0.023 = $11.39/month
• PUTs: 100,000/day × 30 = 3M PUTs × $0.005/1K = $15.00/month
• GETs: 3.9M reads (10% of corpus weekly, see below) × $0.0004/1K = $1.56/month

Total so far: $27.95/month. And look at the shape of it: PUT requests are 54% of the bill. The bytes themselves cost eleven dollars.

This is the part people get wrong. Below roughly a million pages per day, object storage pricing is dominated by request meters, not gigabytes. If your pages average 55 KB compressed, you could store ten times as much data before the GB-month line catches up to the PUT line. Batch your writes — one object holding a hundred pages, newline-delimited — and you cut the PUT meter by 100× at the cost of slightly clumsier reads. For snapshot archives that are only ever reprocessed in bulk, that trade is almost always correct.

The Egress Trap: Reprocessing Pipelines That Re-Bill You for Old Bytes

Here is where the budget explodes. Storage is a one-time-per-byte cost. Egress is not — every pipeline that reads the corpus back out pays the transfer rate again, and again, every time it runs.

Price it for the 495 GB baseline, using the $0.09/GB internet rate:

Reprocessing pattern	Volume pulled monthly	Egress cost
Full corpus re-export, weekly	2,143 GB	$192.87/mo
10% of corpus, weekly	214 GB	$19.27/mo
1% targeted re-parse, weekly	21 GB	$1.93/mo

The weekly full export costs 17× your storage bill. I have seen this exact pattern: a nightly job that syncs the entire snapshot bucket down to a VM “because the parser lives there.” That job is the most expensive component of the whole pipeline, and it is invisible because the charges are labeled as network, not storage.

Fixes, in order of preference:

1. Move compute to the data. Run the parser where the bucket lives. Same-region S3-to-EC2 transfer is free. Cross-region is ~$0.02/GB — still 4.5× cheaper than internet egress.
2. Re-parse only what changed. Conditional GETs with ETags skip unchanged objects entirely.
3. Range requests when you only need a slice of a large object.

The conditional-GET pattern against store.example.com:

import requests

def fetch_if_changed(url: str, etag: str | None):
    headers = {"If-None-Match": etag} if etag else {}
    r = requests.get(url, headers=headers)
    if r.status_code == 304:
        return None, etag          # unchanged — zero bytes transferred
    if r.status_code == 206:       # range hit
        return r.content, r.headers.get("ETag")
    return r.content, r.headers.get("ETag")

# When you only need the head of a large snapshot:
r = requests.get(
    "https://store.example.com/snapshots/2025/...",  # any object key
    headers={"Range": "bytes=0-65535"},
)

The 304 path transfers a response header and nothing else. On a corpus where 90% of pages did not change since the last pass, conditional GETs turn a $193 egress line into roughly $19 — and the GET request meter barely moves, because 304s are billed as cheap reads.

Lifecycle Config That Actually Cuts the Bill: Tiering and Expiring Raw HTML

Retention policy is not a paragraph in a design doc. It is a lifecycle rule, and it should be committed next to the code that writes the objects.

For the baseline — raw HTML you actively re-parse for 30 days, occasionally reference for another 60, and never touch after that:

{
  "Rules": [
    {
      "ID": "snapshots-tier-then-expire",
      "Filter": {"Prefix": "snapshots/"},
      "Status": "Enabled",
      "Transitions": [
        {"Days": 30, "StorageClass": "STANDARD_IA"}
      ],
      "Expiration": {"Days": 90},
      "NoncurrentVersionExpiration": {"NoncurrentDays": 7}
    }
  ]
}

Two details in that policy earn their place. The 30-day transition matches the Infrequent Access minimum storage duration — objects moved to IA before day 30 still bill 30 days of IA, so transitioning earlier is pure loss. And the NoncurrentVersionExpiration matters if your bucket has versioning on; without it, every overwrite leaves a hidden prior version billing you forever.

The before/after math at the baseline volume, 495 GB standing, steady state split 30 days Standard / 60 days IA:

Policy	Storage $/mo	Effective $/GB-mo
All Standard, 90-day retention	$11.39	$0.0230
Tiered (IA at 30d, delete at 90d)	$7.92	$0.0160

A 30% cut. Real, but modest — which is the honest headline. Tiering optimizes the cheapest line item on your bill. Do it, but do not expect it to fix a pipeline that is bleeding money on egress. Also note the IA gotcha: reads from IA carry a per-GB retrieval fee ($0.01/GB on S3). If your reprocessing pass touches the 30-90 day tier heavily, the retrieval fees can eat the tiering savings. Tier cold data that is genuinely cold.

The bigger lever is the expiration date itself. Cutting retention from 90 to 30 days shrinks standing volume by two-thirds, which shrinks every downstream line — storage, egress, GET meters — proportionally.

The Full Ledger: A Recomputable Monthly Budget per Million Pages

Every input is now measured. Time to assemble the whole thing:

def monthly_ledger(pages_per_day, avg_compressed_bytes,
                   retention_days, reprocess_frac_weekly,
                   storage_rate=0.023, put_rate=0.005 / 1000,
                   get_rate=0.0004 / 1000, egress_rate=0.09):

    gb = 1024**3
    standing_gb = pages_per_day * (avg_compressed_bytes / gb) * retention_days
    storage = standing_gb * storage_rate
    puts = pages_per_day * 30 * put_rate

    reads_per_month = pages_per_day * retention_days * reprocess_frac_weekly * 4.33
    gets = reads_per_month * get_rate
    egress = standing_gb * reprocess_frac_weekly * 4.33 * egress_rate

    total = storage + puts + gets + egress
    pages_per_month = pages_per_day * 30
    return {
        "standing_volume_gb": round(standing_gb, 1),
        "storage": round(storage, 2),
        "put_requests": round(puts, 2),
        "get_requests": round(gets, 2),
        "egress": round(egress, 2),
        "total_monthly": round(total, 2),
        "cost_per_1m_pages_fetched": round(total / pages_per_month * 1e6, 2),
    }

print(monthly_ledger(
    pages_per_day=100_000,
    avg_compressed_bytes=55 * 1024,
    retention_days=90,
    reprocess_frac_weekly=0.10,
))

Output for the baseline:

standing_volume_gb:        495.0
storage:                   $11.39
put_requests:              $15.00
get_requests:              $1.56
egress:                    $19.27
total_monthly:             $47.22
cost_per_1m_pages_fetched: $15.74

Fifteen dollars and change per million pages, all-in on the storage side. Compare that to the fetch itself: any managed scraping service bills you dollars per thousand successful pages, so storage lands at roughly half a percent of total pipeline cost at this scale. That ratio is the sanity check — if your storage ledger is a meaningful fraction of your fetch spend, something in the design is wrong, and it is almost always egress or an over-long retention window.

Now the sensitivity table, holding 100K pages/day and 10% weekly reprocessing fixed, varying retention and average compressed page size. Cells are cost per 1M pages fetched:

Avg compressed size	30-day retention	90-day retention	365-day retention
20 KB	$6.40	$9.25	$22.20
55 KB	$8.60	$15.75	$48.60
140 KB	$13.90	$31.55	$112.60

Read the table as a decision surface. Retention is the strongest lever — going from 365 days to 90 cuts the per-page cost by two-thirds at every size. Page size matters less than intuition suggests below 90 days, because the PUT meter (size-independent) is a floor of about $5 per million pages that nothing in the storage tier can remove.

Wrap-Up

The model reduces to five measured inputs: mean page bytes, compression ratio, pages per day, retention days, and reprocessing fraction. Measure the first two yourself — assumed ratios are how budgets end up 3× off — and treat the last three as deliberate policy knobs, not defaults you inherited.

Three claims worth arguing with your team about. First, below a million pages per day, request meters and egress dominate the GB-month line, so batch your writes and keep compute near the bucket before you touch tiering. Second, a weekly full-corpus export can cost more than everything else in the storage budget combined; conditional GETs are the cheapest fix in this entire post. Third, and most contentiously: for retention windows under a couple of weeks, storing snapshots at all is often the wrong call — re-scraping the live page is cheaper than storing, tiering, and re-pulling old bytes, a trade-off explored properly in Re-Scrape vs Change Detection: Choosing a Refresh Strategy.

If you want the fetch side of this ledger modeled with the same rigor, Build a Scrape Cost Ledger: Trace Where Dollars Went covers per-URL attribution. The two ledgers together give you a defensible number for what a page actually costs, end to end — which is the only number that matters when someone asks whether the pipeline pays for itself.