How Retries Quietly Double Your Scraping Costs

Introduction

Your scraping bill went up 40% last month and your delivered page count stayed flat. That’s the retry tax. Every failed attempt costs money — the fetch itself, the bandwidth, the proxy minutes, the JS render — and almost nobody accounts for it when they compute “cost per page.” Teams quote the per-request price the way a store quotes the sticker price, then act surprised when the effective rate is double. This post builds the actual unit economics: attempts-per-success, fallback tiers, proxy depreciation, and the instrumentation you need to see the real number on a dashboard instead of discovering it on an invoice.

The Hidden Math: Why a 90% Success Rate Doubles Your Per-Page Cost

Start with the naive model. If each attempt succeeds independently with probability p, the expected number of attempts to get one success is 1/p. Clean, simple, and wrong in a way that matters — but it’s the baseline everyone has in their head, so let’s compute it.

First-attempt success rate	Expected attempts per success	Cost per delivered page (at $0.001/attempt)	Multiplier
90%	1.11	$0.00111	1.11x
80%	1.25	$0.00125	1.25x
70%	1.43	$0.00143	1.43x

def expected_attempts(p: float) -> float:
    """Naive model: independent Bernoulli attempts."""
    return 1 / p if p > 0 else float("inf")

COST_PER_ATTEMPT = 0.001  # hypothetical price for one example.com fetch

for p in (0.90, 0.80, 0.70):
    print(f"p={p:.0%}  attempts={expected_attempts(p):.2f}  "
          f"cost/delivered=${expected_attempts(p) * COST_PER_ATTEMPT:.5f}")

At 90% success this looks harmless: an 11% overhead. So where does the doubling come from?

Two places. First, failures are not independent coin flips. The 10% of pages that fail once are disproportionately the same pages that fail again — they’re the ones behind an aggressive edge check, or the ones whose selectors break, or the ones served from a region your exit IP handles badly. Your retry attempts run against a much worse success rate than your first attempts. If first attempts hit 90% but retries against the residual hard set only succeed 30% of the time, attempts-per-success climbs from 1.11 toward 1.5, then higher as you dig into the tail.

Second, failed attempts are often more expensive than successful ones. A blocked page frequently means a timeout at full duration, a JS render that burned five seconds of headless browser time before the challenge wall appeared, or a proxy session that got flagged and burned. The failed attempt is the expensive attempt. Multiply a worse success rate by a higher per-attempt cost and 90% headline success can easily land at 2x the sticker price per delivered page.

I laid out the full unit model in What One Successful Page Actually Costs; the rest of this post is about measuring and controlling the retry component of it.

Modeling Attempts-per-Success with a Real Retry Loop

The 1/p formula is a prior, not a measurement. What you actually need is a counter on every fetch that records how many attempts each delivered page consumed. Here’s the loop I start with — plain requests, no framework, one global spend accumulator:

import time
import requests

COST_PER_ATTEMPT = 0.001
spend = 0.0

def fetch_with_retries(url: str, max_attempts: int = 5) -> dict:
    global spend
    attempts = 0
    for i in range(max_attempts):
        attempts += 1
        spend += COST_PER_ATTEMPT
        try:
            r = requests.get(url, timeout=30)
            if r.status_code == 200 and "captcha" not in r.text.lower():
                return {"url": url, "attempts": attempts, "ok": True}
        except requests.RequestException:
            pass
        time.sleep(min(2 ** i, 8))
    return {"url": url, "attempts": attempts, "ok": False}

result = fetch_with_retries("https://example.com/products/1")
delivered = 1 if result["ok"] else 0
print(f"{result}  spend=${spend:.4f}  "
      f"cost/delivered=${spend / max(delivered, 1):.4f}")

Note the response check. A 200 status code is not a delivered page — block pages and interstitials come back as 200 too, and if you count them as successes your success-rate model is fiction from day one. I wrote a whole post about that failure mode: Your 200 OK Responses Are Actually Block Pages.

Run this against a target that’s flaky for you and look at the log:

delivered url=https://example.com/products/1 attempts=4 ok=True
  attempt spend=$0.0040  →  4.0x the per-request sticker price
delivered url=https://example.com/products/2 attempts=1 ok=True
  attempt spend=$0.0050  →  blended 2.5x across 2 pages
failed    url=https://example.com/products/3 attempts=5 ok=False
  attempt spend=$0.0100  →  $0.01 spent, zero pages delivered

Three things jump out of real logs like this. Page 1 took four attempts and cost four times its sticker price on its own. Page 3 cost five attempts and delivered nothing — that spend has to be amortized across every page that did succeed, which is the part most cost models silently drop. And the blended multiplier across just two delivered pages is already 2.5x. Scale that pattern across a million pages and the “cheap” per-request price you negotiated is not the price you’re paying.

If you’re using a scraping API rather than raw requests, the same accounting applies — service-side retries on the FineData scrape endpoint (auto_retry, max_retries) bill tokens per retry, so the multiplier shows up in your usage numbers even though you never wrote a retry loop. Either way, the number you care about is spend divided by delivered pages, nothing else.

Exponential Backoff Is Not Free: Time Costs and Session Burn

Backoff is the standard answer to retry storms, and it’s the right instinct — hammering a target every second after it blocked you just accelerates the block. But backoff has a price column of its own, and it’s mostly invisible because it’s denominated in seconds rather than dollars.

Every second a worker sleeps between attempts is a second that worker slot produces nothing. Every failed attempt that ends in a timeout holds a connection for the full timeout duration. And if you’re using sticky proxy sessions, the backoff window is time your session TTL burns down without producing a page — a session with a 30-minute lifetime that spends 4 of those minutes sleeping through backoff sequences has lost 13% of its capacity before rotation.

Here’s the wall-clock picture for delivering 100 pages from store.example.com, assuming 2-second successful fetches, a 30-second timeout on failed attempts, 10 workers, and 31 total failed attempts spread across the batch:

Strategy	Sleep schedule	Idle sleep per 4-attempt page	Timeout burn per 4-attempt page	Wall-clock, 100 pages / 10 workers
Fixed 1s interval	1s, 1s, 1s	3s	90s (3 failures x 30s)	~116s
Exponential	1s, 2s, 4s	7s	90s	~118s
Exponential, 30s timeout	1s, 2s, 4s	7s	90s	~118s

Read that table honestly: the backoff schedule barely moves wall-clock time, because timeout burn dominates it. The real cost lever isn’t the sleep curve — it’s the timeout and the attempt cap. A retry policy is a cost document whether you meant it to be one or not:

RETRY_POLICY = {
    "max_attempts": 3,      # each extra attempt = one more paid fetch
    "timeout": 30,          # failed attempts burn this in full, per attempt
    "backoff_base": 1,      # seconds; doubles per attempt
    "backoff_cap": 8,       # beyond ~8s, blocks don't lift anyway
    "jitter": True,         # prevents synchronized retry stampedes
}

My opinion, and you may disagree: exponential backoff past about 8 seconds is wasted time for scraping. Backoff curves were designed for transient infrastructure failures — a overloaded service that recovers in seconds to minutes. Anti-bot blocks don’t lift in 16 or 32 seconds. If a target is still failing after two or three attempts with reasonable spacing, more patience won’t convert the attempt into a page; a different exit IP or a different tier will. Cap the backoff low and spend the saved wall-clock on escalation instead. The trade-off discussion in Client-Side vs Service-Side Retries covers where that retry logic should live at all.

Fallback Tiers: When Cheaper Proxies Should Absorb the Failures

If retries against the same infrastructure mostly re-fail, the fix is to stop retrying in place and start escalating. The design: cheap capacity takes the first attempts and absorbs the failures it can, premium capacity fires only on the residue. Most pages are easy. You should not be paying premium prices for the easy 80%.

A three-tier setup against store.example.com, with per-page transfer cost derived from per-GB pricing at ~0.5 MB per page:

Tier	Proxy type	Price	Cost per page (~0.5 MB)	Expected first-attempt success
1	Datacenter	$0.5/GB	$0.00025	60%
2	ISP	$2/GB	$0.001	75%
3	Residential	$8/GB	$0.004	95%

Those success rates are placeholders — you must measure your own per domain, per hour, because they’re the entire input to the cost model. But the shape holds everywhere: datacenter handles the majority, residential exists for the tail.

import requests

API = "https://api.finedata.ai/api/v1/scrape"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}

TIERS = [
    {"name": "datacenter",  "params": {}},                         # baseline
    {"name": "isp",         "params": {"use_isp": True}},          # +2 tokens
    {"name": "residential", "params": {"use_residential": True}},  # +3 tokens
]

def fetch_tiered(attempts_per_tier: int = 2):
    for tier in TIERS:
        for _ in range(attempts_per_tier):
            payload = {
                "url": "https://example.com",
                "formats": ["markdown"],
                **tier["params"],
            }
            r = requests.post(API, json=payload, headers=HEADERS, timeout=130)
            if r.ok:
                data = r.json()
                return {"tier": tier["name"], "data": data}
    return None

page = fetch_tiered()

Two attempts per tier, not five. This is deliberate and it’s the recommendation people push back on: with correlated failures, the third through fifth attempts at the same tier convert at a miserable rate, and each one costs the same as the first. Two attempts catches transient failures — timeouts, blips, a single bad exit IP. Beyond that, the failure isn’t transient, and paying the same price to repeat it is the definition of the retry tax. Escalate instead. The exception is captcha and challenge pages, where a solver needs its attempt budget; that’s a different failure class with its own pricing, covered in Avoid CAPTCHAs vs Solve Them.

Computing the Blended Cost of a Delivered Page

Now combine the tiers into one number. The blended cost per delivered page is the sum, across tiers, of (expected requests at that tier) x (cost per request at that tier), divided by the overall delivery rate.

Using the tier table above, with a 2-2-3 attempt policy:

Step 1 — Tier 1 contribution. Every page starts here: 2 datacenter attempts, always. Cost: 2 x $0.00025 = $0.00050. Delivery rate at this tier: 1 - 0.40² = 84%.

Step 2 — Tier 2 contribution. 16% of pages escalate, running 2 ISP attempts: 0.16 x 2 = 0.32 expected requests. Cost: 0.32 x $0.001 = $0.00032. These pages deliver at 1 - 0.25² = 93.75%, so 0.16 x 0.9375 = 15% of all pages are rescued here.

Step 3 — Tier 3 contribution. 1% of pages reach residential and get 3 attempts: 0.01 x 3 = 0.03 expected requests. Cost: 0.03 x $0.004 = $0.00012. Essentially all of them deliver.

Blended cost per delivered page: $0.00050 + $0.00032 + $0.00012 = $0.00094, at a ~99.999% delivery rate.

Compare that to the lazy alternative — route everything through residential so you never think about tiers:

Strategy	Attempts per delivered page	Cost per delivered page	Cost to deliver 10,000 pages
Residential only	1.05 (at 95% success)	$0.00421	~$42
Tiered (2-2-3)	~2.4	$0.00094	~$9.40

The tiered chain runs more than twice the attempts per page and still costs 4.5x less, because attempts at cheap tiers are nearly free relative to premium ones. This is the whole argument for fallback design in one table: success rate is not the objective function. Cost per delivered page is. A pipeline with a 60% first-attempt success rate and a sane escalation ladder will beat a 95% premium-only pipeline on unit economics every time the easy-page fraction is high, which for most targets it is.

Proxy Burn: Measuring IPs Lost Per 1,000 Requests

Retries don’t just cost request fees. Every retry against a blocking target is another strike against the exit IP that made it, and IP strikes convert into bans. If you run your own pool, retries turn a fixed asset into a depreciating one, and the depreciation rate scales with your attempt multiplier — usually worse than linearly, because retries concentrate on exactly the requests most likely to get an IP flagged.

You can’t manage this without measuring it. Track every request’s exit IP and outcome, then report bans per 1,000 requests:

from collections import defaultdict

requests_by_ip = defaultdict(int)
banned_ips = set()

def record(outcome: str, proxy_ip: str) -> None:
    requests_by_ip[proxy_ip] += 1
    if outcome == "banned":
        banned_ips.add(proxy_ip)

def burn_report(total_requests: int) -> dict:
    return {
        "requests": total_requests,
        "unique_ips_used": len(requests_by_ip),
        "ips_banned": len(banned_ips),
        "bans_per_1k_requests": round(len(banned_ips) / total_requests * 1000, 1),
        "median_requests_per_ip": sorted(requests_by_ip.values())
                                  [len(requests_by_ip) // 2],
    }

Run this across a few thousand requests at different attempt caps and the pattern is consistent: ban counts grow faster than request counts. Illustrative numbers for a 200-IP datacenter pool against example.com, with IP replacement at $2 per IP:

Attempt rate vs. baseline	Requests per 1,000 delivered pages	IPs banned per 1,000 requests	IPs lost per 1,000 delivered	Replacement cost per delivered page
1x (no retries)	1,000	8	8	$0.016
2x	2,000	15	30	$0.060
3x	3,000	25	75	$0.150

The last column is the one to sit with. At a 3x attempt rate, proxy replacement alone costs more per delivered page than everything else in the pipeline combined — and it’s a cost that never appears on a per-request invoice. It shows up three weeks later when your pool is half its size and your success rates collapse, which triggers more retries, which burns more IPs. That feedback loop is how a “small” retry policy change turns into a pipeline incident.

The mitigation is the same as everywhere else: fewer attempts at a failing tier, faster escalation, and a circuit breaker so a blocking event doesn’t grind through your entire pool. Which brings us to the breaker.

Circuit Breakers and Failure Budgets: Capping the Retry Spiral

There’s a failure mode the tier ladder can’t fix: the target is blocking everything, right now, for everyone. Regional outage, aggressive edge rule, your whole IP range got flagged. During that window, every attempt is pure loss, and a retry loop with a 3x multiplier will happily multiply your losses by three while delivering nothing. The circuit breaker exists to make the queue stop.

import time

class CircuitBreaker:
    def __init__(self, threshold: int = 10, cooldown: int = 900):
        self.threshold = threshold      # consecutive failures before tripping
        self.cooldown = cooldown        # seconds before allowing a probe
        self.failures = 0
        self.opened_at = None

    def allow(self) -> bool:
        if self.opened_at is None:
            return True
        if time.time() - self.opened_at >= self.cooldown:
            self.failures = 0           # half-open: reset, let a probe through
            self.opened_at = None
            return True
        return False

    def record(self, ok: bool) -> None:
        if ok:
            self.failures = 0
            return
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = time.time()

breaker = CircuitBreaker(threshold=10, cooldown=900)

def fetch_guarded(url: str) -> dict:
    if not breaker.allow():
        return {"status": "skipped", "reason": "circuit open, cooling down"}
    result = fetch_with_retries(url, max_attempts=1)
    breaker.record(result["ok"])
    return result

Ten consecutive failures trips it; a 15-minute cooldown follows; the next request after cooldown is a probe that either resets the breaker or re-trips it cheaply. While the circuit is open, your workers move on to other domains or idle — either is better than paying to hammer a wall.

The before/after math during a two-hour blocking event on store.example.com, with 10 workers and a $0.001 attempt cost:

	Without breaker	With breaker
Attempts during event	~4,800	~30 (10 to trip, ~20 probes)
Spend during event	$4.80	$0.03
Pages delivered during event	0	0
IPs flagged	dozens	~10
Cost amortized onto later pages	$4.80 / post-event successes	negligible

Zero pages delivered either way — the breaker doesn’t rescue the event. What it does is stop the event from contaminating your unit economics for the next week. Those 4,800 dead attempts don’t vanish; they get amortized into the cost of every page you deliver after the block lifts, silently inflating that domain’s per-page price long after everyone forgot the incident happened. Pair the breaker with a failure budget — “if more than 5% of today’s attempts for this domain failed, halt and alert” — and you’ve turned an unbounded loss into a bounded one.

A Cost Dashboard: Logging Every Attempt to a Per-Page Price Tag

Everything above is model-building. This section is the part that keeps the model honest after you stop paying attention, because the model drifts: targets change their edge rules, your proxy pool degrades, a selector breaks and pages start “failing” at parse time. The only defense is instrumentation that runs permanently and makes the real per-page price visible by domain, tier, and hour.

One SQLite table covers it:

import sqlite3
import time

db = sqlite3.connect("attempts.db")
db.execute("""
    CREATE TABLE IF NOT EXISTS attempts (
        ts      REAL,
        domain  TEXT,
        tier    TEXT,
        cost_usd REAL,
        ok      INTEGER
    )
""")

def log_attempt(domain: str, tier: str, cost_usd: float, ok: bool) -> None:
    db.execute("INSERT INTO attempts VALUES (?,?,?,?,?)",
               (time.time(), domain, tier, cost_usd, int(ok)))
    db.commit()

def cost_per_delivered(domain: str) -> float:
    spend, delivered = db.execute(
        "SELECT SUM(cost_usd), SUM(ok) FROM attempts WHERE domain = ?",
        (domain,),
    ).fetchone()
    return spend / delivered if delivered else float("inf")

Call log_attempt from inside the tiered fetcher — every attempt, success or failure, with its actual tier price. Then a daily rollup gives you the dashboard:

cost per delivered page, last 24h
domain                 tier-1 pages   tier-2   tier-3   $/delivered   trend
store.example.com      18,420         2,910    190      $0.00102      flat
example.com            9,880          410      12       $0.00091      flat
news.example.com       3,100          2,400    1,850    $0.01940      +6x  ← outlier
api.example.com        7,650          0        0        $0.00026      flat

news.example.com is the story here: 60% of its pages are falling through to residential, and its per-page cost is 20x the fleet average. That’s a target whose edge posture changed, and without this table you’d have found out from a monthly invoice review three weeks later. Cross-check the spend side against your provider’s meter — the usage endpoint reconciles your local ledger with what you’re actually being billed:

curl -s -X POST https://api.finedata.ai/api/v1/usage \
  -H "Authorization: Bearer fd_your_api_key"

If the local ledger and the provider’s numbers diverge, you’re missing a cost somewhere — usually JS render tokens or retries you forgot you enabled.

Wrap-Up

Retries are a cost multiplier wearing a reliability costume. The sticker price per request is the beginning of the math, not the end: attempts-per-success compounds, failures cluster on the pages that are hardest to get, failed attempts are frequently the expensive ones, and proxy burn converts a fixed asset into a per-page depreciation expense that never shows up on an invoice.

The working system, assembled from the pieces above:

1. Measure attempts per delivered page, not success rate. Spend divided by delivered pages is the only unit price that matters.
2. Cap attempts per tier at two. The third attempt at a failing tier converts poorly; escalate instead of persisting.
3. Tier your fallbacks so cheap capacity absorbs easy pages and premium capacity only handles the tail. The blended cost beats premium-only by multiples.
4. Circuit-break blocking events so a dead window doesn’t amortize its losses across your next week of successes.
5. Log every attempt to a permanent ledger and review cost-per-delivered-page by domain daily. Regressions are cheap to fix the day they start and expensive three weeks in.

None of this is complicated individually. The failure mode is structural: no single person sees the whole cost chain, because request fees, proxy spend, and worker time live in different budgets. A per-page price tag that rolls all of it into one number fixes the visibility problem, and visibility is most of the fix.