What It Costs to Keep a Page Fresh for a Year
Compute what one monitored page costs per year: fold cadence, retries, render share, and success rate into one recomputable per-page budget number.
On this page
Define the Freshness Contract: Staleness Budget and Check Type for One Page
Every per-page cost argument I’ve ever lost came down to the same root cause: nobody had written down what “fresh” actually means for that page. Freshness is not a vibe. It’s a contract with three clauses — a maximum acceptable data age, the set of fields you compare, and the intensity of the check you’re willing to pay for.
Fix those three things first, because every cost term downstream (cadence, retry volume, render share) is derived from them. A page whose price field can go stale within an hour demands a completely different budget than a page whose promo banner changes twice a quarter.
Here’s the contract I use, expressed as config:
# freshness_contract.yaml
page:
url: https://store.example.com/products/widget-42
staleness_budget: 45m # max acceptable age of price data
fields_to_compare:
- price
- stock_status
- promo_badge
check_type: http_first # plain GET by default, render on escalation
change_signal: content_hash # sha256 of normalized field payload
on_change:
- recheck_after: 10m # confirm the change stuck (not a flap)
- notify: pricing-team
Note the on_change recheck. A single observed change is not a fact — A/B tests, cache layers, and canary deploys all produce phantom flips. Budget for the confirmation check or your change log will be full of noise.
Different page classes on the same site deserve different contracts:
| Page class (store.example.com) | Staleness budget | Fields compared | Check intensity |
|---|---|---|---|
Product page (/products/widget-42) | 45 min | price, stock, promo badge | HTTP poll, render on hash delta |
Category listing (/category/tools) | 6 h | product set, rank order | HTTP poll, daily full render |
| Promo banner (site-wide) | 24 h | banner text, target URL | One render per day |
The product page is where the money is, literally, so it gets the tight budget and the expensive escalation path. The category listing tolerates hours of staleness because rank order rarely matters intraday. The banner gets one render a day because it’s JavaScript-injected and cheap to check rarely.
If you take one thing from this section: refuse to compute a monitoring budget for anyone who can’t fill in this YAML. “Monitor it continuously” is not a contract.
The Per-Page Cost Formula: Folding Cadence, Retries, Render Share, and Success Rate into One Number
Once the contract exists, the annual cost of honoring it is four variables and some arithmetic. Here’s the whole model:
# yearly_checks: raw poll volume, inflated by retries
yearly_checks = checks_per_hour * 8760 * (1 + retry_multiplier)
# base_cost: every check billed at the cheap plain-HTTP rate
base_cost = yearly_checks * base_check_cost
# render_cost: the fraction of checks that escalate to a headless render,
# priced at the premium over the plain check
render_cost = yearly_checks * render_share * render_premium
# failure term: failed checks still bill, and sustained failures add
# alert fan-out plus human time; modeled as a fixed annual pool
# scaled by the failure fraction
total = base_cost + render_cost + failure_overhead * (1 - success_rate)
Four knobs. That’s it. Cadence sets the volume, the retry multiplier inflates it, render share decides how much of that volume is expensive, and success rate prices the tail of checks that fail outright.
The same thing in Python, runnable as-is:
def budget_page(
checks_per_hour: float, # from the freshness contract
retry_multiplier: float, # derived from your poll log (next sections)
base_check_cost: float, # $ per plain-HTTP check
render_share: float, # 0.0 - 1.0, fraction of checks that render
render_premium: float, # $ premium of a render over a plain check
success_rate: float, # 0.0 - 1.0, fraction of checks that succeed
failure_overhead: float, # $ annual pool for alerts + human time
) -> dict:
yearly_checks = checks_per_hour * 8760 * (1 + retry_multiplier)
base_cost = yearly_checks * base_check_cost
render_cost = yearly_checks * render_share * render_premium
failure_cost = failure_overhead * (1 - success_rate)
return {
"yearly_checks": round(yearly_checks),
"base_cost": round(base_cost, 2),
"render_cost": round(render_cost, 2),
"failure_cost": round(failure_cost, 2),
"total": round(base_cost + render_cost + failure_cost, 2),
}
# Example: https://example.com monitored at a 30-minute cadence
print(budget_page(
checks_per_hour=2.0,
retry_multiplier=0.09,
base_check_cost=0.0005,
render_share=0.20,
render_premium=0.002,
success_rate=0.97,
failure_overhead=40.0,
))
The point of compressing everything into one function is that it becomes recomputable. When a constant drifts — and it will — you re-run the function, not the argument. This is the same unit-economics discipline as cost per successful page, applied to a single monitored URL over a year instead of a crawl.
Deriving Fold Cadence: Sizing the Check Interval from Observed Change Frequency
The most common budget error is picking cadence by intuition. Measure it instead. Poll the page aggressively for a month, log every content hash, and build a change-frequency histogram. Then fold back to a cadence that matches reality.
Here’s what 30 days of monitoring the price field on https://store.example.com/products/widget-42 produced from my poll log:
price-field change events per day, 30-day sample (N = 41 events)
0 changes | day 19 | ████████████████████████████
1 change | day 7 | ███████████
2 changes | day 3 | ████
3+ changes | day 1 | █
Median inter-change gap: 14 hours. Ninety-fifth percentile gap: 38 hours. The 3+ day was a site-wide promo flip, not a pricing decision. So the question becomes: what does a 15-minute cadence buy over a 60-minute cadence for a page that changes once or twice a day? Almost nothing — except in the worst case, where you learn about a change up to an hour later.
| Cadence | Checks/year (pre-retry) | Worst-case staleness | Relative annual cost |
|---|---|---|---|
| 5 min | 105,120 | 5 min | 8.0x |
| 15 min | 35,040 | 15 min | 2.7x |
| 60 min | 8,760 | 60 min | 1.0x |
My opinion, and you may disagree: for most product pages, 60 minutes is the right cadence, and the 8x premium for 5-minute polling buys you a freshness improvement nobody’s downstream decision actually consumes. The exception is a page inside a competitive repricing loop — if your system reacts to a competitor’s price within minutes, the staleness budget is genuinely 15 minutes and you pay. Otherwise you’re buying precision you never read.
If you already have a poll log, one line gets you the per-day change counts:
jq -r 'select(.changed == true) | .ts[0:10]' poll_log.jsonl | sort | uniq -c
Run that before you commit to a cadence. It takes five minutes and it’s the highest-leverage five minutes in this whole exercise.
Pricing the Retry Tail: Backoff Policy and the Retry Multiplier
Not every check succeeds on the first attempt. Timeouts happen, a proxy exit gets a soft block, the origin returns a challenge page. Your retry policy turns each of those into extra billed checks, and the formula needs one number to represent it: the retry multiplier.
The policy for store.example.com checks:
retry_policy:
max_attempts: 3
backoff: exponential # 10s -> 40s -> 160s
retry_on: [timeout, 5xx, block_page_signature]
do_not_retry_on: [4xx_except_429]
The backoff spacing matters less for cost than for success — what matters for cost is the average number of retries per failed check. From the monitor log: 6% of checks fail their first attempt, and failed checks average 1.5 retries before resolving (some succeed on retry one, some burn all three). The multiplier falls out:
retry_multiplier = failure_rate * avg_retries_per_failure
= 0.06 * 1.5
= 0.09 # i.e., 1.09x the nominal check volume
One subtlety worth calling out: retries per failure are rarely geometric in practice. If the first attempt failed because your exit IP got flagged, the retry through a fresh IP usually succeeds — but if the failure came from an origin-side incident, retries fail in clusters and burn the full cap. That clustering is exactly why the empirical average (1.5) is higher than a naive per-attempt model would predict. Don’t derive this number analytically; pull it from logs.
A quick simulation to confirm the multiplier holds:
import random
random.seed(42)
N = 10_000
FAIL_RATE = 0.06
RETRY_DIST = [0.60, 0.30, 0.10] # P(1 retry), P(2), P(3) -- from poll log
total_attempts = 0
for _ in range(N):
attempts = 1
if random.random() < FAIL_RATE:
attempts += random.choices([1, 2, 3], weights=RETRY_DIST)[0]
total_attempts += attempts
print(f"multiplier: {total_attempts / N:.3f}") # ~1.09
Run it with your own logged distribution before plugging the multiplier into the budget. If your retry policy is client-side, also read up on the trade-off between client-side and service-side retries — who owns the retry changes who pays for it.
Render Share: Splitting Cheap HTTP Polls from Expensive Headless Renders
Every check is not equal. A plain HTTP GET with a browser-like TLS fingerprint costs a fraction of a full headless render, and for most monitored pages the plain check answers the question “did anything change?” just fine. The trick is to make rendering the exception.
| Check type (https://example.com) | What it costs | What it catches |
|---|---|---|
| Plain HTTP GET | base rate (~$0.0005/check) | Server-rendered HTML changes, status flips |
| Headless render | base + premium (~$0.0025/check, 5x) | JS-injected content, client-side hydration, lazy data |
The escalation pattern: poll cheap, render only when the cheap poll detects something. If the plain GET’s content hash differs from the last known-good hash, escalate to a render to see the full picture — because a hash delta on the raw HTML might be a real change or might be a template tweak that a render would resolve identically to last time.
escalation_rule:
target: https://store.example.com/products/widget-42
poll:
interval: 5m
method: http # plain GET, use_antibot stays on
escalate_when:
content_hash_changed: true
escalation:
use_js_render: true # headless render, +tokens per check
js_wait_for: networkidle
target_render_share: 0.20 # budget guardrail: alert if exceeded
And if you route the checks through a scraping API, the same split applies — the render call just adds use_js_render: true to the request body, and the token delta between the two call shapes is your render premium, observable directly in the usage and balance numbers.
To calibrate the premium against your own infrastructure instead of a provider’s price sheet, time a local render:
# calibrate_render_premium.py -- run once, record the delta
import time
from playwright.sync_api import sync_playwright
URL = "https://example.com"
t0 = time.perf_counter()
# ... plain HTTP GET via your normal client ...
http_s = time.perf_counter() - t0
with sync_playwright() as p:
browser = p.chromium.launch()
t0 = time.perf_counter()
page = browser.new_page()
page.goto(URL, wait_until="networkidle")
_ = page.content()
render_s = time.perf_counter() - t0
browser.close()
print(f"http: {http_s:.2f}s render: {render_s:.2f}s ratio: {render_s/http_s:.1f}x")
That ratio — typically 10-20x on wall-clock time, which maps to a 3-6x cost premium once you amortize the browser fleet — is the number to defend. If your render share creeps from 20% to 50%, your per-page cost roughly doubles, and nothing about the freshness contract changed. Guardrail the share, don’t just observe it.
Success Rate as a Multiplier: Billed Failures, Alerts, and Incident Overhead
Here’s the part most budgets miss: a failed check is not a free check. If 3% of your checks fail, you still paid for them — the request went out, tokens or compute were consumed, and then you paid again for the retries. And beyond the billed attempts, failures generate a second cost stream: alert fan-out, dashboards going red, and eventually a human opening a terminal.
Worked example at 97% success rate for store.example.com/products/widget-42, at a 30-minute cadence:
- Nominal checks: 17,520/year. With the 1.09 retry multiplier: ~19,097 billed attempts.
- The failing 3% still bills its checks and its retries — that’s already inside the 19,097.
- On top of that: each sustained outage window (a failure that survives all retries) triggers alert fan-out. At the observed rate of roughly one sustained window per month, assume 15 minutes of human triage per window — about 3 hours of engineer time per year, plus pager noise. Priced conservatively at a $40 annual pool per page, scaled by the failure fraction:
failure_cost = 40.00 * (1 - 0.97) = $1.20/year per page
That looks trivial for one page. It isn’t. You’re not monitoring one page — you’re monitoring thousands, and the per-page failure cost multiplies with the fleet while the human overhead doesn’t scale down. A 95% fleet-wide success rate means a sustained-failure window somewhere every few days, and your on-call rotation feels it.
| Success rate | Added annual cost per page (failure term) | What it means operationally |
|---|---|---|
| 99% | $0.40 | Roughly one triage per quarter |
| 97% | $1.20 | Roughly one triage per month |
| 95% | $2.00 | Roughly one triage every 1-2 weeks |
The honest trade-off: chasing 99% success usually means upgrading proxy quality or stealth level on every check, which raises base_check_cost for all checks to fix a 3% tail. Sometimes that’s correct. Often it’s cheaper to keep the cheap checks cheap and spend only on the failure path — escalate to residential proxies or stronger stealth on retry rather than on the first attempt.
Worked Budget: Twelve Months of Monitoring store.example.com/products/widget-42
Assemble every constant from the previous sections into one run:
#!/usr/bin/env python3
"""Annual freshness budget for one monitored product page."""
# --- Freshness contract (section 1) ---
CHECKS_PER_HOUR = 2.0 # 30-min cadence, fits the 45m staleness budget
# --- Retry tail (section 4) ---
RETRY_MULTIPLIER = 0.09 # 6% failure rate, 1.5 avg retries per failure
# --- Check pricing (section 5) ---
BASE_CHECK_COST = 0.0005 # $ per plain-HTTP check
RENDER_SHARE = 0.20 # escalate on hash delta only
RENDER_PREMIUM = 0.002 # $ premium per render over plain check
# --- Failure overhead (section 6) ---
SUCCESS_RATE = 0.97
FAILURE_OVERHEAD = 40.0 # $ annual alert + human pool per page
def budget_page():
yearly_checks = CHECKS_PER_HOUR * 8760 * (1 + RETRY_MULTIPLIER)
base_cost = yearly_checks * BASE_CHECK_COST
render_cost = yearly_checks * RENDER_SHARE * RENDER_PREMIUM
failure_cost = FAILURE_OVERHEAD * (1 - SUCCESS_RATE)
total = base_cost + render_cost + failure_cost
print(f"yearly billed checks : {yearly_checks:,.0f}")
print(f"base poll cost : ${base_cost:.2f}")
print(f"render cost : ${render_cost:.2f}")
print(f"failure overhead : ${failure_cost:.2f}")
print(f"TOTAL PER PAGE/YEAR : ${total:.2f}")
budget_page()
# yearly billed checks : 19,097
# base poll cost : $9.55
# render cost : $7.64
# failure overhead : $1.20
# TOTAL PER PAGE/YEAR : $18.39
Call it $18.40 per page per year. Roughly half of that is the plain polling volume, forty percent is the render escalation, and the rest is failure tax.
Now the part that actually changes decisions — sensitivity. Vary cadence and render share, hold everything else constant:
| Render share 10% | Render share 25% | Render share 50% | |
|---|---|---|---|
| 15-min cadence | $27.94 | $39.39 | $58.49 |
| 30-min cadence | $14.57 | $20.30 | $29.85 |
| 60-min cadence | $7.88 | $10.17 | $15.52 |
Read the corners. The cheapest cell and the most expensive cell differ by 7.4x — same page, same contract fields, same success rate. Cadence dominates, but render share is the multiplier that silently doubles you. A team that “just monitors every 15 minutes to be safe” while letting 50% of checks escalate to renders is paying $58 per page; the team that measured change frequency and disciplined its escalation pays $15. Across a 10,000-page fleet that’s the difference between $585K and $155K a year.
Sanity check the figure against a real price list before you trust it. Pull your provider’s plan pricing and compute the per-check rate for both call shapes — plain GET versus a request with use_js_render: true — then substitute those numbers for BASE_CHECK_COST and RENDER_PREMIUM and re-run. If your provider prices per token, the plain check with default use_antibot is the base rate and the render adds its token surcharge; the arithmetic is identical. The point of the model is that the provider is a swappable constant, not the frame.
Recompute Triggers: Keeping the Budget Number Honest Over Time
$18.40 is a snapshot. The page will get heavier, the provider will change prices, the render share will drift. A budget number that was honest in January is fiction by June unless you define what invalidates it. Here are the triggers I use:
| Trigger | Threshold | Why it invalidates the budget |
|---|---|---|
| Page weight change | > 30% median bytes over 30 days | Heavier pages cost more to fetch and render; may also shift render share |
| Provider price change | Any change to per-check or per-token pricing | Directly rescales BASE_CHECK_COST / RENDER_PREMIUM |
| Render share drift | > 5 percentage points from plan (e.g., 20% → 25%) | Render cost is linear in share; 5 points is ~$1/year/page |
| Success rate drop | Below 95% sustained for 2 weeks | Retry multiplier and failure overhead both understate reality |
| Cadence renegotiation | Freshness contract changes | Everything downstream changes |
The last row is the one people forget. When the product team decides price data now matters intraday, the contract YAML changes, and the budget must be recomputed and re-approved — not silently absorbed.
Automation for the recompute, as a systemd timer pair:
# /etc/systemd/system/budget-recompute.service
[Unit]
Description=Recompute per-page freshness budget and flag drift
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
WorkingDirectory=/opt/monitoring
ExecStart=/opt/monitoring/venv/bin/python recompute_budget.py \
--page https://store.example.com/products/widget-42 \
--config freshness_contract.yaml \
--drift-alert 0.15
# /etc/systemd/system/budget-recompute.timer
[Unit]
Description=Monthly budget recompute
[Timer]
OnCalendar=monthly
Persistent=true
[Install]
WantedBy=timers.target
The --drift-alert 0.15 flag means: if the recomputed annual total moves more than 15% from the last recorded figure, page someone. A scheduled GitHub Actions job with a cron trigger works just as well if you’d rather keep this in CI next to the code that owns the constants.
Wrap-Up
One page, one year, one number: about $18.40, built from four honest constants — cadence from a measured change histogram, a retry multiplier pulled from real poll logs, a render share guarded by an escalation rule, and a success rate that prices the failure tail including the human part. The formula is small on purpose. Small formulas get recomputed; sprawling cost models get ignored.
Three things to take away. First, the freshness contract precedes the math — refuse to budget a page whose staleness limit and compared fields haven’t been written down. Second, cadence is the dominant term, and most teams pay 3-8x more than their decision loop requires; measure change frequency before choosing an interval. Third, render share is the silent doubler — the escalation guardrail matters more than any proxy tuning.
Recompute monthly, alert on drift, and treat the per-page figure as a living constant rather than an annual surprise. When you’re ready to scale this from one page to a fleet, the same function runs per URL class — and the aggregate becomes a forecast you can defend line by line.
Related Articles
Build a Scrape Cost Ledger: Trace Where Dollars Went
Log a cost dimension on every fetch and build a per-URL ledger that shows where your scraping budget actually goes — renders, retries, and wasted pages.
TechnicalToken Economics of LLM Extraction: A Cost Worksheet
Measure the real token cost of LLM extraction: tokens per page, per field, and per clean record, with a worksheet you can recompute on your own corpus.
TechnicalBudget the Fetches Behind Every Data Row
Every data row rides on listing pages, pagination, and dead links. Model the fetch-to-row ratio, plug in real numbers, and budget crawls you can defend.