Build a Scrape Cost Ledger: Trace Where Dollars Went

Why Raw Request Counts Lie: Modeling the True Cost of a Single Fetch

Ask most teams what a scrape costs and they’ll divide monthly spend by pages fetched. That number is wrong in a specific, expensive way: it treats every fetch as identical. In practice, one fetch of store.example.com/product/123 with JavaScript rendering, two retries, and a residential proxy can cost 8-10x a plain HTTP GET that succeeds on the first attempt. Averaging them together hides exactly the URLs you need to see.

A fetch’s cost is a composite. Render time is compute, billed by the second. Bytes transferred are bandwidth. Retries multiply everything before them — and if you’re paying per attempt rather than per successful page, retries are pure loss. Proxy tiering adds a multiplier on top: datacenter is cheap, residential costs more, mobile carrier exits cost more still. And a 403 that returned a 12KB block page still consumed a proxy slot, bandwidth, and your rate budget.

Here’s a decomposition for one realistic fetch:

Line item	Value	Notes
Base request	2 tokens	Plain GET, TLS fingerprint only
JS render	+5 tokens	Headless browser, 4.2s render
Residential proxy	+3 tokens	Needed for this domain
Retry #1 (timeout)	+10 tokens	Full render cost repeated
Retry #2 (success)	+10 tokens	Rendered again, succeeded
Bandwidth	340KB total	Includes the failed attempt’s partial body
Effective cost	30 tokens, 2 attempts	vs. 2 tokens for the naive model

The ledger view says this URL cost 15x what your average suggests. That’s the number that should drive whether you keep crawling it.

The fix starts with a data model. Every fetch gets a cost record with explicit dimensions — not a single scalar:

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class CostDimension:
    url: str
    crawl_run_id: str
    attempt: int
    status_code: int
    render_ms: int = 0
    bytes_transferred: int = 0
    retry_count: int = 0
    proxy_cost_micros: int = 0   # micros = 1e-6 dollars, avoids float drift
    fetch_cost_micros: int = 0
    ts: datetime = field(default_factory=datetime.utcnow)

Micros as integers is deliberate. Floating-point dollar amounts accumulate rounding error across millions of rows, and integer micro-dollars sort, sum, and compare cleanly. If your provider bills in tokens or credits, keep a second column for that unit — never convert to dollars until you report, because pricing changes and you’ll want to re-derive history.

Instrumenting Every Fetch: Wrapping Your HTTP Client with a Cost Recorder

Cost data that lives outside the fetch path rots. The recorder has to sit inside the client wrapper, capturing every attempt — including the ones that never return a usable body. If instrumentation is a separate step someone forgets to call, you’ll have a ledger that only covers the happy path, which is worse than no ledger because it lies with authority.

For plain HTTP fetches, wrap the session with a response hook:

import time
import httpx

class CostRecordingClient:
    def __init__(self, ledger, crawl_run_id):
        self.ledger = ledger
        self.run_id = crawl_run_id
        self.client = httpx.Client(timeout=30)

    def get(self, url, **kwargs):
        start = time.monotonic()
        try:
            resp = self.client.get(url, **kwargs)
            elapsed_ms = int((time.monotonic() - start) * 1000)
            self.ledger.append(CostDimension(
                url=url,
                crawl_run_id=self.run_id,
                attempt=kwargs.get("_attempt", 1),
                status_code=resp.status_code,
                render_ms=0,
                bytes_transferred=len(resp.content),
            ))
            return resp
        except httpx.TimeoutException:
            elapsed_ms = int((time.monotonic() - start) * 1000)
            self.ledger.append(CostDimension(
                url=url,
                crawl_run_id=self.run_id,
                attempt=kwargs.get("_attempt", 1),
                status_code=-1,  # timeout, no response
                render_ms=elapsed_ms,
            ))
            raise

Note the timeout branch. A timed-out fetch is often the most expensive fetch you make — you burned the full timeout window of compute and a proxy slot and got nothing. If your recorder only fires on successful responses, timeouts vanish from the ledger and your per-URL costs for slow pages look artificially healthy.

Rendered fetches need the same treatment with different dimensions. When you submit a render job through a scraping API, the response metadata is where the cost signals live — render time, whether JS was on, which proxy tier ran. A wrapper for a rendered fetch of store.example.com looks like:

def scrape_rendered(url, ledger, run_id):
    payload = {
        "url": url,
        "formats": ["markdown"],
        "use_js_render": True,
        "use_residential": True,
        "only_main_content": True,
    }
    start = time.monotonic()
    resp = requests.post(
        "https://api.finedata.ai/api/v1/scrape",
        json=payload,
        headers={"Authorization": "Bearer fd_your_api_key"},
        timeout=120,
    )
    elapsed = int((time.monotonic() - start) * 1000)
    body = resp.json()
    ledger.append(CostDimension(
        url=url,
        crawl_run_id=run_id,
        attempt=1,
        status_code=resp.status_code,
        render_ms=elapsed,
        bytes_transferred=len(resp.content),
        fetch_cost_micros=estimate_cost_micros(payload),
    ))
    return body

The estimate_cost_micros function is just a lookup: base price plus per-feature surcharges (JS render, residential, captcha solve), derived from your provider’s published token costs. Hardcode it in config, not in the wrapper, so repricing is a one-line change.

One trade-off worth stating plainly: client-side wall-clock timing includes your own network latency and queue waits, so it overstates render cost somewhat. That’s fine. You’re comparing URLs against each other within the same run, not billing anyone. Consistent measurement beats precise measurement here.

Attributing Retries and Failure Costs to the URL That Caused Them

Retries are where budgets go to die quietly. The standard pattern — retry with backoff, log the eventual outcome — charges the whole retry chain to the crawl’s average. The URL that needed five attempts looks identical to the one that succeeded instantly. Your ledger exists to break that.

The rule is simple: every attempt writes its own cost entry, keyed by URL and attempt number. The retry loop becomes:

def fetch_with_retry(client, url, max_retries=5):
    for attempt in range(1, max_retries + 1):
        try:
            resp = client.get(url, _attempt=attempt)
            if resp.status_code == 200 and resp.content:
                return resp
            if resp.status_code in (403, 429):
                time.sleep(backoff(attempt))
                continue
            return resp  # 404 etc. -- don't retry, but the cost is already logged
        except (httpx.TimeoutException, httpx.TransportError):
            if attempt == max_retries:
                raise
            time.sleep(backoff(attempt))
    return None

Because the wrapper records on every pass through the loop — including the exception paths — the ledger sees all five attempts. No extra bookkeeping in the retry logic itself.

The difference attribution makes is stark. Same crawl, two accounting methods:

URL	Unattributed (spread into average)	Attributed	Delta
store.example.com/product/123	$0.004 (run average)	$0.004	—
store.example.com/product/456	$0.004	$0.004	—
store.example.com/search?q=widget	$0.004	$0.021	5.25x
store.example.com/product/789	$0.004	$0.004	—

That search endpoint with its 403-retry loop was consuming real money, and the average priced it at a fifth of its true cost. This is the core argument for per-URL attribution over per-run metrics, and it’s the same logic behind how retries quietly double your scraping costs — except here you catch it per URL instead of per crawl.

An honest caveat: attribution by URL can mislead when failures are caused by your proxy pool rather than the target. If a bad proxy exit poisons 50 URLs in a row, per-URL accounting flags 50 “expensive” URLs when the real culprit is one IP. Always slice the ledger by exit IP or proxy profile alongside URL before pruning anything — more on that in the action section.

Storing the Ledger: A Per-URL Cost Schema You Can Query

An append-only list of cost events is a good start, but analysis needs aggregation, and aggregation needs a schema you can GROUP BY against. SQLite is the right default here — single file, zero ops, and it handles tens of millions of rows without complaint. Don’t reach for a time-series database until you’ve felt SQLite’s limits personally.

Two tables: a raw event log (what the wrapper appends) and a rollup per URL per run:

CREATE TABLE cost_events (
    id INTEGER PRIMARY KEY,
    ts TEXT NOT NULL,
    crawl_run_id TEXT NOT NULL,
    url TEXT NOT NULL,
    attempt INTEGER NOT NULL,
    status_code INTEGER NOT NULL,
    render_ms INTEGER NOT NULL,
    bytes_transferred INTEGER NOT NULL,
    fetch_cost_micros INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_events_run_url ON cost_events(crawl_run_id, url);

CREATE TABLE cost_ledger (
    crawl_run_id TEXT NOT NULL,
    url TEXT NOT NULL,
    fetch_count INTEGER NOT NULL,
    render_ms_total INTEGER NOT NULL,
    bytes_total INTEGER NOT NULL,
    retries INTEGER NOT NULL,
    cost_micros_total INTEGER NOT NULL,
    PRIMARY KEY (crawl_run_id, url)
);

The rollup is a materialized view you recompute with a single INSERT ... SELECT ... GROUP BY after each run (or incrementally during it). Keeping both means you can answer “what did this run cost per URL” instantly, while retaining the event log for questions you haven’t thought of yet — like “which status codes dominate retries on this domain,” which needs attempt-level rows.

Writes should be batched. One executemany per few hundred events, not one insert per fetch:

import sqlite3

class LedgerWriter:
    def __init__(self, path="ledger.db"):
        self.conn = sqlite3.connect(path)
        self.buffer = []

    def append(self, dim: CostDimension):
        self.buffer.append((
            dim.ts.isoformat(), dim.crawl_run_id, dim.url,
            dim.attempt, dim.status_code, dim.render_ms,
            dim.bytes_transferred, dim.fetch_cost_micros,
        ))
        if len(self.buffer) >= 250:
            self.flush()

    def flush(self):
        self.conn.executemany(
            "INSERT INTO cost_events (ts, crawl_run_id, url, attempt, "
            "status_code, render_ms, bytes_transferred, fetch_cost_micros) "
            "VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
            self.buffer,
        )
        self.conn.commit()
        self.buffer.clear()

Flush on process exit and on a timer, not just on buffer size — a crashed crawl that loses its last 250 cost events has a hole in its accounting, and holes make the ledger untrustworthy.

Computing Cost per Successful Page: Separating Productive from Wasted Fetches

Total spend is a vanity metric. The number that matters is cost per successful page — dollars divided by pages that actually yielded extracted items. That requires joining the cost ledger against an extraction-success table, because a fetch can return HTTP 200 with a block page and still be a total waste. If you haven’t internalized that failure mode, read why your 200 OK responses are actually block pages — a non-trivial share of “successful” fetches in any large crawl deliver nothing.

The join:

def cost_per_useful_page(conn, crawl_run_id):
    rows = conn.execute("""
        SELECT l.url,
               l.cost_micros_total,
               l.fetch_count,
               COUNT(s.url) AS useful_pages
        FROM cost_ledger l
        LEFT JOIN extraction_success s
          ON s.url = l.url AND s.crawl_run_id = l.crawl_run_id
        WHERE l.crawl_run_id = ?
        GROUP BY l.url
    """, (crawl_run_id,)).fetchall()
    return rows

From this join you derive three metrics I consider non-negotiable for any pipeline that runs daily:

• Cost per useful page: run-level total cost divided by count of URLs with at least one extracted item.
• Wasted-fetch ratio: attempts on URLs that never yielded data, divided by total attempts.
• Per-URL excess: cost of a URL minus the run’s median cost per useful page.

A sample ledger output makes this concrete. Three URLs from a store.example.com run:

url                                        attempts  cost_micros  useful  excess
store.example.com/product/123              1         4_100        yes     —
store.example.com/product/456              1         4_100        yes     —
store.example.com/search?q=widget          7         34_800       no      30_700
run totals: 43_000 micros, 2 useful pages
cost per useful page: 21_500 micros

One URL consumed 81% of the run and produced nothing. Without the ledger, this run reports as “3 pages fetched, ~$0.014 per page, all fine.” With it, you see a search endpoint that’s structurally hostile to your current fetch configuration — and you fix or drop it.

Here’s a position some people will disagree with: I think cost per useful page should gate your retry policy, not success rate. A URL at 40% success rate that costs 2 tokens per attempt is fine to keep retrying. A URL at 90% success rate that requires premium stealth and a residential proxy might be your worst page economically. Success rate measures resilience; cost per useful page measures value. Optimize the second, monitor the first.

Visualizing Where Dollars Went: Grouping Ledger Data by URL, Domain, and Failure Reason

Once the ledger exists, the analysis is plain SQL. Two aggregations cover most of what you need day to day.

Cost by domain, to see which targets eat the budget:

SELECT
    substr(url, 1, instr(substr(url, 9), '/') + 8) AS domain,
    SUM(cost_micros_total) AS total_micros,
    SUM(retries) AS total_retries,
    COUNT(*) AS urls
FROM cost_ledger
WHERE crawl_run_id = ?
GROUP BY domain
ORDER BY total_micros DESC;

Cost by failure reason, derived from the event log’s status codes and body sizes:

SELECT
    CASE
        WHEN status_code = -1 THEN 'timeout'
        WHEN status_code = 403 THEN '403_blocked'
        WHEN status_code = 429 THEN 'rate_limited'
        WHEN status_code = 200 AND bytes_transferred < 2048 THEN 'empty_body'
        ELSE 'ok_' || status_code
    END AS failure_reason,
    COUNT(*) AS events,
    SUM(fetch_cost_micros) AS cost_micros
FROM cost_events
WHERE crawl_run_id = ?
GROUP BY failure_reason
ORDER BY cost_micros DESC;

The empty_body case deserves a comment: a 200 with a sub-2KB body on a product page is almost always a block page or a JS-only shell. Classifying it as a failure in the query — not as success — is what keeps the wasted-fetch ratio honest.

A console report from a real-shaped run:

Top 5 most expensive URLs — run crawl_8f3a, store.example.com
rank  url                                   cost_micros  drivers
1     /search?q=widget                     34_800        6 retries, residential, js_render
2     /product/999                         11_200        stealth_antibot, 2 retries
3     /product/123                         4_100        baseline
4     /product/456                         4_100        baseline
5     /category/electronics                3_900        js_render, 1 retry

Domain rollup: store.example.com 58_100 micros (97% of run)
Failure rollup: 403_blocked 41_200 micros, timeout 6_400, empty_body 2_900

You don’t need a dashboard to start. A cron job that prints this to Slack or a log file changes behavior within a week, because the expensive URLs stop being invisible. Dashboards come later, once you know which slice you actually look at.

Acting on the Ledger: Budget Caps, URL Pruning, and Retry Policy Tuning

A ledger that only gets admired is overhead. Three controls convert it into a system that defends its own budget.

First, per-domain caps enforced by consulting the ledger before each fetch:

budget_caps:
  store.example.com:
    max_micros_per_run: 500_000   # $0.50
    action: skip_and_alert
  example.com:
    max_micros_per_run: 2_000_000
    action: skip

def under_budget(conn, url, run_id, caps):
    domain = extract_domain(url)
    cap = caps.get(domain)
    if not cap:
        return True
    spent = conn.execute(
        "SELECT COALESCE(SUM(cost_micros_total), 0) "
        "FROM cost_ledger WHERE crawl_run_id = ? AND url LIKE ?",
        (run_id, f"https://{domain}%"),
    ).fetchone()[0]
    return spent < cap["max_micros_per_run"]

This is a circuit breaker with a dollar-denominated trip condition — a natural companion to circuit breakers triggered by target degradation, but keyed on spend rather than error rates, which catches the “everything succeeds but each success costs 20 tokens” failure mode that error-based breakers never trip on.

Second, URL pruning based on trailing cost. Before the scheduler pops a URL, check its recent history and drop the expensive ones:

def should_prune(conn, url, window=5, threshold_micros=15_000):
    rows = conn.execute("""
        SELECT cost_micros_total FROM cost_ledger
        WHERE url = ? ORDER BY crawl_run_id DESC LIMIT ?
    """, (url, window)).fetchall()
    if len(rows) < window:
        return False  # not enough history — give it a chance
    avg = sum(r[0] for r in rows) / window
    return avg > threshold_micros

The len(rows) < window guard matters. Pruning on thin history kills new URLs before they’ve had a fair shot, and a domain that changed its bot defenses last Tuesday will look expensive for exactly the window it takes your config to adapt. Prune on sustained cost, not one bad run.

Third, retry tuning driven by observed per-URL cost. The ledger tells you which URLs burn retries on 403s — those need a different fetch configuration (a residential exit, a stronger stealth tier, or a country-matched proxy), not more retries. Retrying the same configuration against the same defense is paying repeatedly for a result you’ve already proven you can’t get that way. Cap retries at 2 for URLs whose trailing average exceeds your threshold, and spend the saved budget on a config change instead.

One last opinion: build the ledger before you build anything else in your cost stack. Teams instinctively reach for a better proxy pool or a cheaper provider first, but without attribution you can’t measure whether either change helped. The ledger is a weekend of work, runs on a machine you already own, and every optimization after it becomes measurable instead of anecdotal.

Wrap-Up

Cost observability for scraping is not a finance problem — it’s a pipeline design problem. Model each fetch as a composite of render, bandwidth, retries, and proxy cost. Record every attempt, including timeouts and block pages, at the point of the request. Attribute retries to the URL that caused them, roll events into a queryable per-URL ledger, and join it against extraction success so you’re measuring cost per useful page rather than cost per attempt. Then close the loop: budget caps per domain, pruning rules for chronically expensive URLs, and retry policies informed by what actually fails and why.

The pattern generalizes, too. The same ledger answers latency questions, proxy-quality questions, and extraction-quality questions — the dimensions you’re already recording are the raw material for all of them. Start with SQLite, start this week, and let the first “top 5 most expensive URLs” report tell you where your budget actually went.