What a Proxy Hour Really Costs: A Burn Budget
Model the real cost of proxying a scrape: bandwidth per page, rotation overhead, and failure burn you can recompute from your own request logs.
On this page
The Unit Mismatch: Why Proxy Invoices Lie
Proxy invoices lie to you. Not deliberately — they just report in units that have nothing to do with the unit you actually care about. You buy gigabytes, IP-hours, or requests, but the only thing your pipeline produces is successfully fetched pages. Everything else is overhead, and the gap between the two is where budgets quietly double.
This post builds a burn budget: a cost model for proxying a crawl that you can recompute from your own request logs, with no vendor marketing numbers anywhere in it.
Cost Per Successful 200: Reframing Proxy Pricing Into One Comparable Unit
Take the same 250 KB product page fetched through three common pricing models. Assume a 10% failure rate, so each successful page takes about 1.11 attempts on average. The failure multiplier is applied consistently in every row — including the IP-hour row, where the throttle ceiling is stated in attempts and divided through to successes:
| Pricing model | List price | Assumptions | Cost per successful 200 |
|---|---|---|---|
| Per-GB | $0.50/GB | 250 KB billed per attempt (uncompressed — see next section), 1.11 attempts/success | $0.000139 |
| Per-request | $0.0005/request | Billed per attempt, 1.11 attempts/success | $0.000556 |
| Per-IP-hour | $0.02/IP-hour | 400 attempts/hour sustained per IP × 1.11 → 360 successes/hour | $0.000056 |
Same page. Same target. A 10x spread between the per-request and per-IP-hour rows, purely as a function of how the vendor slices the meter. Per-request pricing punishes you for small pages; per-GB punishes you for big ones — or for pools that bill the uncompressed body, which is most of them; per-IP-hour punishes you for idle and underutilized capacity, which is the fixed_fees term and gets its own treatment at the end of this post.
One byte-counting caveat before you quote any of this: the per-GB row above bills 250 KB of raw HTML per attempt. The next section shows the same page is ~62.5 KB on the wire when gzip is negotiated. If your pool meters the compressed wire payload instead of the raw body, that row drops 4x to $0.000035. Which figure the pool meters is a question worth asking in writing before you compare quotes — it’s a 4x swing on the headline number.
The only way to compare quotes — or to sanity-check the quote you already have — is to convert everything into one unit:
cost_per_200 = (bytes_paid_per_attempt
× attempts_per_success
× price_per_byte)
+ fixed_fees
Three terms, and the rest of this post takes them apart in turn. bytes_paid_per_attempt is bigger than you think (headers, handshakes, uncompressed bodies). attempts_per_success is a function of your failure rate and retry policy, and it behaves in a way most people misjudge. fixed_fees is where per-IP-hour pools hide their costs, because an idle-but-reserved IP burns money whether it fetches or not.
If you take one habit from this post, make it this: never quote a proxy price without dividing it through to cost per successful 200. A “cheap” $0.50/GB pool that serves you full-size 200-OK block pages three times out of four costs $0.00050 per delivered page — more than a $1.50/GB pool with a 98% success rate at $0.00038. I’ve seen teams switch pools to save 20% on the sticker price and lose 40% to failure burn.
Counting the Bytes: Headers, TLS Handshake, and the True Payload of a store.example.com Page
The bytes you’re billed for are not the bytes your parser sees. Every fetch is a stack, and every layer of the stack has a meter running:
your request line + headers (~800 B, billed as egress)
→ TCP SYN + TLS ClientHello (~500 B)
→ server cert chain + response (~4,000 B for a typical chain)
→ response headers (~1,200 B)
→ compressed body (the part you actually wanted)
Measure the gap on a real page before you model anything. Two curl runs against the same URL, with and without compression:
curl -s -o /dev/null \
-w 'download=%{size_download} headers=%{size_header}\n' \
https://store.example.com/product/123
curl -s -o /dev/null --compressed \
-w 'download=%{size_download} headers=%{size_header}\n' \
https://store.example.com/product/123
For a representative 250 KB product page, here’s what the stack looks like:
| Component | Bytes | Notes |
|---|---|---|
| Raw HTML | 250,000 | What your parser sees; what the first table billed |
| HTML gzipped on the wire | 62,500 | ~4:1 is typical for markup |
| Response headers | 1,200 | Cookies, CSP, cache directives |
| TLS handshake, fresh connection | 4,000 | ClientHello plus cert chain, both directions |
| Your request, upstream | 800 | Many pools bill egress only; some bill both ways |
Two things jump out. First, the gzip line: markup compresses roughly 4:1, and whether the proxy gateway negotiates Accept-Encoding: gzip is a 4x swing in your bandwidth bill. This is the reconciliation with the first table — the $0.000139 row assumes the pool meters the 250 KB raw body; a pool that meters the 62.5 KB wire payload bills $0.000035 for the same page. Check whether your pool strips that header. Some do, because uncompressed responses are more billable.
Second, the handshake line is 6% of the compressed payload — but only if you open a fresh connection per request. With keep-alive it amortizes across hundreds of requests. Under aggressive rotation, it never amortizes. Which brings us to rotation.
Rotation Overhead: Handshake Tax, Warmup Requests, and When Sticky Sessions Pay Off
Rotation is not free, and its cost is not measured in bytes. Every fresh exit IP means a fresh TCP connection, a fresh TLS handshake, and — on targets that fingerprint visitors — a cold session with no cookies and no history. Cold sessions also pay warmup requests: a robots.txt fetch, a favicon, a redirect hop to the regional storefront. Every one of those is a billed attempt that returns no data. The target sees a brand-new visitor every time, and brand-new visitors get scrutinized.
Here’s a harness that quantifies the latency half of that tax. The FineData scrape API used below is async-only — there is no synchronous scrape endpoint — and that matters for measurement: if you timed only the POST /async/scrape round-trip, you’d be measuring how fast the API accepts a job, not how long the page takes to fetch, and the comparison would be worthless. So the honest number is submission-to-terminal latency: submit a job, poll GET /api/v1/async/jobs/{job_id} every 0.5 s until the job settles into a terminal state (completed, failed, or cancelled), and clock the whole span. One guard keeps fast failures from flattering the average: any response that isn’t checked for status and body — an instant 429 or 403, say — is skipped and counted as an error, never timed:
import statistics, time, requests
BASE = "https://api.finedata.ai/api/v1"
HDRS = {"Authorization": "Bearer fd_your_api_key"}
TERMINAL = {"completed", "failed", "cancelled"} # pending/processing are not terminal
def timed_batch(label, extra, n=100):
lat, errors = [], 0
for _ in range(n):
t0 = time.perf_counter() # clock starts at submission
r = requests.post(f"{BASE}/async/scrape", headers=HDRS, timeout=60,
json={"url": "https://example.com",
"formats": ["markdown"],
**extra})
if r.status_code != 200 or not r.text:
errors += 1 # fast 429/403: skipped, never timed
continue
job_id = r.json()["id"]
while time.perf_counter() - t0 < 600: # 10-minute cap per job
s = requests.get(f"{BASE}/async/jobs/{job_id}",
headers=HDRS, timeout=60)
if s.status_code != 200 or not s.text:
errors += 1 # status poll failed: skip this job
break
if s.json()["status"] in TERMINAL:
lat.append(time.perf_counter() - t0) # submission → terminal state
break
time.sleep(0.5)
else:
errors += 1 # never reached a terminal state
if not lat:
print(f"{label}: no measurable jobs, errors={errors}")
return
lat.sort()
print(f"{label}: ok={len(lat)} errors={errors} "
f"median={statistics.median(lat):.2f}s "
f"p95={lat[min(94, len(lat) - 1)]:.2f}s")
timed_batch("rotating", {})
The API above manages rotation on its side and exposes no session-pinning parameter, so there’s nothing to wire into extra for a pinned run — when your pool does expose one, run the harness twice and diff the distributions. The sticky row below is the shape that diff takes on a pool where pinning is configurable:
| Policy | Median latency | p95 | Pages/min/worker | Hours to finish 50k pages (20 workers) |
|---|---|---|---|---|
| Rotating | 1.42s | 2.9s | 42 | 59.5 |
| Sticky (10-min session) | 1.18s | 2.1s | 51 | 49.0 |
Roughly ten extra wall-clock hours on a 50k crawl, from nothing but repeated handshakes and cold connections. If you’re paying for infrastructure by the hour — worker containers, orchestrator slots, your own time waiting on a backfill — rotation overhead is real money that never shows up on the proxy invoice.
The counter-argument matters, though, and I want to be honest about it: sticky sessions concentrate your traffic on fewer IPs, which raises per-IP request rates and can push block rates up. A sticky session is a bet that the target tolerates repeat visits from one IP better than it tolerates a parade of strangers. For most e-commerce targets, that bet pays. For targets with per-IP rate limits, it doesn’t. Measure both, per target — the harness above is a few dozen lines, there’s no excuse for guessing.
Modeling Failure Burn: Retries, CAPTCHA Walls, and Bytes You Pay For Twice
Here is the part most cost models get wrong. A failed attempt is not a zero-cost event. You paid for the request bytes, the handshake, the response headers, and the body — which on a block page is often a 2-5 KB “access denied” stub, or worse, a full page that returns HTTP 200 while containing none of your data. If you haven’t read about that failure mode, see Your 200 OK Responses Are Actually Block Pages — it inflates this section’s numbers more than you’d expect, because a blocked-but-200 page looks like a success in your logs while contributing nothing. And if the failure is a CAPTCHA wall and you’re paying a solver per attempt, that surcharge rides the same multiplier as your bytes do.
The math for retries is a geometric series. With failure probability f and a cap of r retries (n = r + 1 total attempts):
def expected_cost_per_200(bytes_per_attempt, price_per_byte,
failure_rate, retry_cap):
n = retry_cap + 1 # total attempts allowed
attempts = (1 - failure_rate ** n) / (1 - failure_rate)
p_success = 1 - failure_rate ** n # P(URL ever succeeds)
return attempts / p_success * bytes_per_attempt * price_per_byte
Now the non-obvious result. attempts / p_success simplifies to exactly 1 / (1 - failure_rate) — for every retry cap. The cap doesn’t change what a successful page costs. It only changes how many URLs you give up on:
| Failure rate | Cap 1 (mult / abandoned) | Cap 2 | Cap 3 | Cap 5 |
|---|---|---|---|---|
| 10% | 1.11 / 1.0% | 1.11 / 0.1% | 1.11 / 0.01% | 1.11 / ~0% |
| 30% | 1.43 / 9.0% | 1.43 / 2.7% | 1.43 / 0.8% | 1.43 / 0.02% |
| 50% | 2.00 / 25% | 2.00 / 12.5% | 2.00 / 6.3% | 2.00 / 1.6% |
Read any row left to right and the multiplier never moves; the abandonment rate collapses. The retry cap is a coverage dial, not a unit-cost dial. Two consequences follow. First, your cost per delivered page is set entirely by the failure rate: at $0.50/GB with 250 KB billed per attempt, a 10% failure rate costs $0.000139 per 200, a 30% rate costs $0.000179, and a 50% rate costs $0.000250 — same pool, same sticker price. Second, abandoned URLs are pure dead burn. At a 30% failure rate with cap 1, you give up on 9% of your target list, and every abandoned URL has already consumed two full attempts of billable bytes that will never be attributed to any delivered page. That’s the “bytes you pay for twice” — really, bytes you pay for and never get.
The Third Term: fixed_fees and Idle Reserved-IP Burn
Per-IP-hour pools don’t bill fetches; they bill reservations. That $0.000056 figure in the first table is a floor, not a price — it assumes every reserved IP sustains 400 attempts per hour, every hour, for the full duration of the reservation. Define utilization u as the successes you actually deliver divided by the capacity you reserved. Your real cost per 200 is:
effective_cost_per_200 = $0.000056 / u
Idle burn comes for u from more directions than people expect: targets whose per-IP rate limits sit below your 400/hour ceiling, politeness delays, bursty crawl windows that leave IPs reserved but starved between runs, backoff storms during block events, and the long tail of a crawl when the queue thins but the reservations persist. Run the numbers:
| Utilization | Effective cost per 200 | vs. per-GB ($0.000139) | vs. per-request ($0.000556) |
|---|---|---|---|
| 100% | $0.000056 | 2.5x cheaper | 10x cheaper |
| 50% | $0.000111 | cheaper | 5x cheaper |
| 40% | $0.000139 | break-even | cheaper |
| 25% | $0.000222 | 1.6x more expensive | still cheaper |
| 10% | $0.000556 | 4x more expensive | break-even |
The pool that looked 10x cheaper at full utilization ties the per-request pool at 10% utilization and stops beating even the conservative per-GB number at 40%. Metered pools set fixed_fees to zero; reservation pools turn the entire invoice into the fixed term — which is why utilization, not list price, is the number to negotiate.
Wrap-Up: The Whole Formula, Populated
Put the running example through all three terms:
cost_per_200 = bytes_paid_per_attempt × attempts_per_success × price_per_byte + fixed_fees
Per-GB pool, bills uncompressed body:
250,000 B × 1.11 × $0.50/GB + $0 ≈ $0.000139 per delivered page
Per-GB pool, meters the gzip wire payload:
62,500 B × 1.11 × $0.50/GB + $0 ≈ $0.000035 per delivered page
Per-request pool:
1.11 attempts × $0.0005/request ≈ $0.000556 per delivered page
Per-IP-hour pool at utilization u:
$0.02 / (360 × u) + $0 = $0.000056 at u = 1.0
= $0.000111 at u = 0.5
Three inputs decide everything: what your pool actually meters (raw body or wire payload — a 4x question), your true failure rate from request logs (not the vendor’s SLA page), and the utilization of anything you’ve reserved. Recompute the formula from your own logs monthly, because failure rates drift with the target’s defenses and utilization drifts with your schedule. The invoice will keep reporting gigabytes, IP-hours, and requests. You should keep reporting one number: cost per successful 200.
Related Articles
How Retries Quietly Double Your Scraping Costs
Failed attempts still cost money. Model attempts-per-success, fallback tiers, and proxy burn to compute what a delivered page truly costs.
TechnicalForecast Scrape Spend: A Budget Model You Can Recompute
Turn success rates, retry multipliers, and rendering overhead into a scraping budget you can defend, with formulas and worked numbers you can recompute.
TechnicalWhat One Successful Page Actually Costs: A Unit Model
Build a per-page cost model for scraping: retries, rendering, proxy traffic, and CAPTCHA solving, with worked numbers you can recompute for your own budget.