Amortize Maintenance Into Your Per-Page Cost

Why Per-Page Pricing Silently Understates Your True Costs

Ask an engineer what a scraped page costs and you’ll get an answer in fractions of a cent. Proxy spend divided by successful pages. API tokens divided by successful pages. Maybe compute gets a line item if someone is feeling thorough. The number looks scientific. It is also wrong, usually by a factor of two to four at moderate scale.

What’s missing is the time nobody invoices for: the Tuesday afternoon you spent fixing a selector that broke when a product page template changed. The morning you burned figuring out why pagination suddenly returns empty results past page 30. The hours debugging retry logic after a target started soft-blocking your exit IPs. None of that shows up in a token meter or a proxy invoice, but all of it is a real cost of producing those pages.

Here is the worked calculation that usually ends the argument. Take 60 hours of selector fixes over a year — a conservative figure for a single mid-sized target — at a $75/hr blended rate. That is $4,500. Spread across 50 million pages scraped in the same period, it comes to $0.00009 per page. Trivial, right?

That calculation is the trap. First, selector fixes are one category out of five or six; total maintenance for a real pipeline runs two to four times that. Second, 50M pages is a large-scale operation — at 5M pages the same 60 hours works out to $0.0009 per page, and stacking that on top of a naive $0.001 metered figure puts you at roughly $0.0019 per page before you have counted a single other maintenance category. That one line item alone nearly doubles the quoted cost. Third, volume assumptions decay. When a target’s crawlable surface shrinks or you deliberately cut refresh frequency, the denominator drops while the maintenance numerator keeps ticking.

The comparison across project sizes makes the distortion concrete:

Project size	Pages/year	Naive per-page cost	Maintenance hours/year	Maintenance per page	Fully-loaded per-page
Small (3 targets)	2M	$0.0010	80 ($6,000)	$0.0030	$0.0040
Mid (15 targets)	20M	$0.0010	160 ($12,000)	$0.0006	$0.0016
Large (60 targets)	100M	$0.0010	260 ($19,500)	$0.0002	$0.0012

Two things jump out. Maintenance amortizes with volume, so small projects get hit hardest — the exact projects where nobody bothers to model costs. And at small scale, maintenance is three times the naive cost, meaning every pricing or margin decision made on the naive number is off by 4x.

If you want the companion argument about how retries inflate the metered side of this equation, see How Retries Quietly Double Your Scraping Costs. This piece covers the other half: the labor nobody meters.

Cataloging Your Maintenance Burden: A Taxonomy of Scraper Repairs

You cannot amortize what you cannot name. “Maintenance” as a single line item is useless because it mixes five different failure modes with different frequencies, different costs, and different fixes. Break it apart first.

Category	What breaks	Typical frequency	Hours per incident
Selector drift	CSS/XPath selectors stop matching after template tweaks	1–2x per month per active target	2–6
Pagination changes	URL schemes, infinite scroll offsets, or page caps shift	Roughly quarterly per target	1–3
Anti-bot countermeasures	Target deploys new checks; block pages return 200 OK	Weekly to monthly on hostile targets	3–8
Data schema shifts	Fields rename, move, or vanish from the page structure	Quarterly, often batched with redesigns	2–5
Infrastructure failures	Job queue stalls, disk fills, scheduler misses, socket leaks	Monthly across a fleet	1–4

The frequencies are estimates from typical operations, not laws — a target that redesigns twice a year will blow through the selector drift row. The point is that each row is separately countable, and once you count them, two things happen: you stop treating maintenance as an amorphous time sink, and you discover which categories actually dominate. Most teams I have walked through this exercise are surprised that anti-bot countermeasures and selector drift together account for well over half of total repair hours.

The habit starts with a single function call. Every time you sit down to fix something, log it:

import json, datetime

def log_repair(domain, category, hours, root_cause, notes=""):
    event = {
        "ts": datetime.datetime.utcnow().isoformat(),
        "domain": domain,             # "example.com"
        "category": category,         # "selector_drift"
        "hours": hours,               # 3.5
        "root_cause": root_cause,     # "product template v2 removed .price-current"
        "notes": notes,
    }
    with open("repairs.jsonl", "a") as f:
        f.write(json.dumps(event) + "\n")

Yes, it is a manual entry. That is fine — repairs are discrete events, not continuous telemetry, and a human logs them in thirty seconds. The discipline matters more than the tooling.

Instrumenting Repairs: Logging Maintenance Hours Against Target Domains

Manual logging captures the incident. But some maintenance is invisible precisely because it happens inside normal operation: a parser that silently degrades, a selector that matches 60% of pages instead of 100%, a fallback path that quietly runs on every request. You instrument for that.

The pattern I use is a decorator around each domain’s parser function. It times execution, counts field-level failures, and appends everything to a SQLite table. When fill rates drop or a parser starts throwing, you have a timestamped record of when degradation began — which is usually the moment maintenance started accruing, even if nobody noticed for a week.

import functools, sqlite3, time, datetime, json

DB = sqlite3.connect("parser_telemetry.db")
DB.execute("""CREATE TABLE IF NOT EXISTS parse_runs (
    ts TEXT, domain TEXT, duration_ms REAL,
    ok INTEGER, missing_fields TEXT)""")

def instrumented(domain):
    def wrap(fn):
        @functools.wraps(fn)
        def inner(html, *args, **kwargs):
            t0 = time.monotonic()
            try:
                result = fn(html, *args, **kwargs)
                missing = [k for k, v in result.items() if v is None]
                ok = 1 if not missing else 0
                return result
            except Exception:
                ok, missing, result = 0, [], None
                raise
            finally:
                dur = (time.monotonic() - t0) * 1000
                # Telemetry must never mask the parser's own exception:
                # swallow DB errors here and let the original error propagate.
                try:
                    DB.execute(
                        "INSERT INTO parse_runs VALUES (?,?,?,?,?)",
                        (datetime.datetime.utcnow().isoformat(),
                         domain, dur, ok, json.dumps(missing)))
                    DB.commit()
                except sqlite3.Error:
                    pass
        return inner
    return wrap

@instrumented("store.example.com")
def parse_product(html):
    # ... selectors, extraction, normalization ...
    return {"title": None, "price": None}  # stub

Two details are load-bearing. The except branch sets missing to an empty list, not the string "[]", so json.dumps always produces the same JSON array shape in the missing_fields column — a consistent schema matters the first time you try to query it. And the database write inside finally is wrapped in its own try/except: if the telemetry database is locked or the disk is full, the original parser exception still reaches your error handler instead of being replaced by a sqlite3.OperationalError.

Pair this with the manual repair log and you get both halves of the picture. The telemetry tells you when and how often things degraded. The repair log tells you what it cost to fix them. Joined on domain and date, they produce something rare: a defensible maintenance-hours-per-domain number.

A slice of what the combined record looks like for store.example.com:

2025-...  # (timestamps omitted for brevity)
repair  store.example.com  selector_drift    2.0h  ".price-current removed from template"
repair  store.example.com  pagination        1.5h  "offset param renamed, page cap at 40"
repair  store.example.com  anti_bot          4.0h  "200 OK block pages on datacenter IPs"
repair  store.example.com  schema_shift      3.0h  "availability field moved to JSON-LD"

One honest caveat: instrumentation itself is maintenance. The decorator commits one SQLite write per parse, which is fine at moderate volume but will become a bottleneck at high parse rates — at that point, buffer rows in memory and flush in batches, or move the writes to a background thread. Keep it deliberately dumb until then — one table, no ORM, no dashboard until the raw data has proven useful for a full quarter.

Choosing an Amortization Window: Monthly, Quarterly, or Rolling Average

Once you have measured hours, you must decide what period to spread them over. This choice shapes your pricing behavior more than the underlying data does.

The formula is simple either way:

maintenance_per_page = (hours_in_window × blended_rate) / pages_in_window

The one rule that matters: the hours and the pages must come from the same window. Annualize both or monthly-ize both; mixing an annual numerator with a monthly denominator (or dividing by the window length a second time) quietly deflates your maintenance cost by whatever factor you mismatched, and the model will happily confirm whatever you already believed.

Short windows (one to three months) are reactive. A redesign in month two immediately reprices months three through five. That is correct if you are billing a customer and need to pass costs through quickly — but it also means your quoted per-page rate whipsaws, and a single bad month can make a healthy target look terminal.

Long windows (twelve-month rolling) are stable. The same redesign gets smoothed across a year of volume. That is better for internal budgeting and for deciding whether a target is worth keeping at all, because one incident does not dominate the average. The cost is lag: a target that genuinely got harder takes up to a year to fully reprice.

Run the same dataset both ways and the difference is stark. store.example.com normally needs about 6 maintenance hours per month at 400k pages. A redesign lands and that month costs 36 hours.

Window	Hours captured	Pages covered	Maintenance per page
3-month (incl. redesign month)	48	1.2M	$0.0030
12-month rolling	102	4.8M	$0.0016

The 3-month window nearly doubles the apparent cost of the target. The 12-month window barely moves. Neither is wrong — they answer different questions. “What do I charge next quarter?” wants the short window. “Do I keep scraping this domain?” wants the long one.

My recommendation, and you may disagree: use a 12-month rolling average for all internal cost accounting and target-keep/drop decisions, and only drop to a 3-month window when you are pricing a pass-through contract. Teams that price off short windows tend to churn targets after every redesign, which is usually the wrong call — redesign-driven maintenance spikes decay, while the underlying value of the data often does not.

Computing the Fully-Loaded Per-Page Cost: A Worked Model

Now assemble everything. The model takes three inputs per domain, all measured over the same window: maintenance hours, page volume, and infrastructure spend split into compute and proxy/API lines. Output is one number you can actually make decisions with.

BLENDED_RATE = 75.0  # $/hour, fully-loaded engineering cost

def fully_loaded_per_page(domain, maint_hours, pages, compute_spend,
                          proxy_spend):
    """All inputs are totals over the SAME window (here: 12 months).
    maint_hours:   e.g. 14 h/month * 12 = 168
    pages:         e.g. 500_000/month * 12 = 6_000_000
    compute_spend: e.g. $300/month * 12 = $3_600
    proxy_spend:   e.g. $450/month * 12 = $5_400
    """
    maint_cost = (maint_hours * BLENDED_RATE) / pages
    compute    = compute_spend / pages
    proxy      = proxy_spend / pages
    return {
        "domain": domain,
        "maintenance_per_page": round(maint_cost, 6),
        "compute_per_page": round(compute, 6),
        "proxy_per_page": round(proxy, 6),
        "total_per_page": round(maint_cost + compute + proxy, 6),
    }

print(fully_loaded_per_page(
    domain="example.com",
    maint_hours=14 * 12,     # 14 h/month, annualized
    pages=500_000 * 12,      # 500k pages/month, annualized
    compute_spend=300 * 12,  # $300/month compute
    proxy_spend=450 * 12,    # $450/month proxies + scraping API
))

Because the caller annualizes every input, the function divides by pages once and only once — no separate window_months term. That is the whole trick: the window is already baked into both numerator and denominator, and re-dividing by it is how models end up reporting costs 12x lower than reality.

For example.com at 500k pages/month, 14 maintenance hours/month, $300 compute and $450 proxy spend:

Cost line	Monthly spend	Per page
Compute	$300	$0.0006
Proxies / scraping API	$450	$0.0009
Amortized maintenance (14h × $75)	$1,050	$0.0021
Total	$1,800	$0.0036

Maintenance is 58% of the total. More than compute and proxies combined. If you had quoted this target at a naive $0.0015 per page, you would be losing money on every page and feeling good about the volume.

If you route scraping through a managed API, the token spend sits in the proxy line and the usage endpoint gives you the denominator directly — GET /api/v1/usage returns balance and consumption, and GET /api/v1/user/stats with date_from/date_to parameters gives per-period volume you can join against your repair log. No estimation required for the volume side.

For the deeper treatment of unit economics — including retry costs and success-rate effects on the metered lines — What One Successful Page Actually Costs: A Unit Model covers the metered half in detail.

Weighting Costs by Domain Volatility

A flat per-page rate across all your targets is a subsidy scheme: stable domains pay for volatile ones. Two targets with identical page volume can differ by 5x in maintenance hours, and a single blended rate hides that completely.

The fix is a volatility index. Define it as repair hours per 10,000 pages over your rolling window:

volatility_index = (repair_hours / pages) × 10_000

Compute it per domain:

• example.com: 168 hours/year over 6M pages = 0.28 h/10k pages
• store.example.com: 72 hours/year over 4.8M pages = 0.15 h/10k pages

Pick a baseline — say 0.10 h/10k pages for a well-behaved target — and derive multipliers from it. Feed those into the cost model as a per-domain surcharge on the maintenance line, or simply track them so pricing conversations start from the right number:

VOLATILITY = {
    # domain: multiplier applied to baseline maintenance cost
    "example.com":        2.8,   # 0.28 h/10k pages vs 0.10 baseline
    "store.example.com":  1.5,   # 0.15 h/10k pages
    "api.example.com":    0.4,   # JSON endpoint, near-zero DOM maintenance
}

Note the third entry, and this is the opinion part: targets with a stable JSON endpoint deserve a discount, not just parity. If api.example.com exposes clean JSON, its DOM maintenance is near zero and its per-page cost collapses. The strategic implication is that endpoint discovery is a cost-reduction activity, not just an engineering preference. I would rather spend four hours hunting for a hidden JSON endpoint than spend four hours every quarter re-fixing selectors — the endpoint work amortizes to zero, the selector work never does. The trade-off, covered in Hidden JSON Endpoints vs DOM Parsing: What to Scrape, is that undocumented endpoints can vanish without notice, so keep the DOM parser as a fallback and monitor both.

Setting Alert Thresholds When Maintenance Cost Drifts Upward

A cost model that nobody monitors decays into a spreadsheet museum. The failure mode is gradual: a target gets 5% harder each month, nobody notices, and eighteen months later the margin is gone. You need a tripwire.

The threshold I use: flag any domain whose rolling maintenance-per-page exceeds 40% of its fully-loaded per-page cost. At that point maintenance is no longer overhead — it is the dominant cost line, and the target’s economics deserve an explicit decision rather than inertia.

def check_thresholds(domains):
    alerts = []
    for d in domains:
        maint = d["maintenance_per_page"]
        total = d["total_per_page"]
        share = maint / total
        if share > 0.40:
            alerts.append(
                f"ALERT {d['domain']}: maintenance is "
                f"{share:.0%} of total (${maint:.4f} of ${total:.4f} per page)"
            )
    return alerts

domains = [
    {"domain": "example.com",
     "maintenance_per_page": 0.0021, "total_per_page": 0.0036},
    {"domain": "store.example.com",
     "maintenance_per_page": 0.0009, "total_per_page": 0.0024},
]
print("\n".join(check_thresholds(domains)))
# ALERT example.com: maintenance is 58% of total ($0.0021 of $0.0036 per page)

example.com trips the wire immediately. That is not a bug in the threshold — it is the model telling you this target was mispriced from the start. The decision table for breaches:

Breach severity	Response
40–55% of total, single cause (e.g. selector drift)	Fix the extraction approach: switch to schema-driven extraction or a JSON endpoint; recheck next quarter
40–55%, multiple recurring causes	Reprice the target or reduce refresh frequency; check whether change detection can replace full re-scrapes
55–70%, trending upward two quarters	Cache more aggressively, cut page scope to the fields that actually earn revenue
>70% or trending past 70%	Drop the target, or renegotiate if it is a paid data contract

The middle rows are where most of the value lives. Reducing refresh frequency is underused: if maintenance is driven by scraping 500k pages/month but only 50k of those pages ever change, you are paying repair hours for data you throw away. Change detection versus full re-scrape is its own trade-off, but at high maintenance shares it almost always wins.

Revisiting the Model Quarterly: A Maintenance Cost Review Checklist

The model is a snapshot. Targets change, volumes shift, and last quarter’s volatility weights go stale. Put a quarterly review on the calendar — an hour, not a day — and run through this:

Review item	What to do	Red flag
Re-measure repair hours	Pull the last 3 months from the repair log per domain	Any domain with zero logged hours — you forgot to log, or you stopped scraping
Re-verify page volumes	Join API usage stats and your own crawl counts	Volume down >20% while hours flat — per-page cost is rising silently
Recalculate volatility weights	Recompute h/10k pages, update multipliers	A domain crossing 2x its prior index
Compare predicted vs. actual cost	Last quarter’s model output vs. actual spend	Drift >15% means a model input is wrong, not that reality is wrong
Audit the instrumentation	Confirm the parse_runs table is still filling	Gaps in telemetry — the decorator got removed in a refactor

Here is what a real review outcome looks like. example.com was modeled at $0.0036 per page. A redesign mid-quarter drove maintenance from 14 to 25 hours per month — an extraction escalation, the kind of event where you climb from CSS selectors to schema-driven extraction to a heavier engine, each rung costing more time. The recomputed model:

Cost line	Before redesign	After redesign
Compute	$0.0006	$0.0006
Proxies / API	$0.0009	$0.0009
Amortized maintenance	$0.0021	$0.0037
Total	$0.0036	$0.0052

Maintenance share went from 58% to 71%. The decision table says: drop, renegotiate, or restructure. In this case the right call was restructuring — cutting refresh from weekly to daily-diff-only, which halved page volume and pushed the maintenance share back under the threshold, at the cost of slightly staler data. That trade was acceptable because the downstream consumer only acted on price changes, not on every snapshot. If they had needed full freshness, the honest answer would have been repricing the data contract upward by roughly 40%, and being able to show the worked model is what makes that conversation winnable instead of hand-wavy.

The Number You Quote Is Not the Number You Pay

Naive per-page cost is a metered number; true per-page cost is a metered number plus labor. The gap is largest exactly where people least expect it — small projects, volatile domains, and post-redesign quarters. The full playbook fits in five moves: categorize repairs so they are countable, instrument parsers so degradation is timestamped, amortize over a 12-month rolling window for internal decisions, weight by per-domain volatility instead of blending flat, and trip an alert when maintenance crosses 40% of total cost.

The uncomfortable conclusion is that some of your targets are unprofitable at their current price and refresh rate, and have been for a while. The model does not fix that. It just makes it visible — and visibility is the difference between a pricing decision and a slow margin leak.