Client-Side vs Service-Side Retries for Scraping Failures

Why Retry Handling Decides Your Scrape Budget

Retrying a failed scrape sounds trivial until you count what a retry actually costs. Every attempt burns proxy bandwidth, adds wall-clock latency, and — if you own the retry loop — occupies a worker thread or a queue slot while it sleeps through a backoff delay. Meanwhile, most scraping APIs will retry internally if you ask them to, and the pricing model determines whether those internal attempts cost you anything.

The question is not “should I retry failed scrapes.” Obviously yes. The question is where the retry lives: in your code, inside the service, or split between both. Each answer has a different failure profile, a different cost curve, and a different operational burden. This comparison walks through all three.

What Actually Fails: Classifying Scrape Errors Before Choosing a Retry Strategy

Before writing any retry code, sort your failures into classes. Retrying a request that can never succeed wastes money; not retrying one that would have succeeded on the second attempt wastes data. Here is the taxonomy I use:

Error class	Typical cause	Retryable client-side?	Retryable service-side?	Verdict
Timeout	Slow origin, oversized page, JS render hang	Yes, with longer timeout	Yes, often with a different proxy	Retryable
HTTP 403	IP blocked, TLS fingerprint flagged	Only with a new exit IP	Yes, service rotates IP + fingerprint	Retryable service-side only
HTTP 429	Rate limiting	Yes, with backoff + lower concurrency	Yes, service throttles and re-queues	Retryable
HTTP 503	Origin overloaded, temporary	Yes	Yes	Retryable
Connection reset	Middlebox interference, proxy death	Yes, immediately	Yes	Retryable
HTTP 404	Page genuinely gone	No	No	Non-retryable
Empty/missing parse	Page loaded but content didn’t match selector	Yes, but only after fixing the selector	No — retrying won’t fix a bad selector	Fix the code, don’t retry

The distinction that matters most: 403s and hard blocks are almost never fixable by re-sending the same request from the same IP. A naive client-side retry loop that hammers a blocked URL five times makes the block worse and can get your IP range flagged permanently. A service-side retry can swap exit IPs and TLS fingerprints between attempts, which is a categorically different retry.

Here’s what three distinct failure signatures look like in logs from a crawl of https://example.com/products:

[INFO]  GET https://example.com/products/sku-1042 -> 200 OK (1.8s, 142KB)
[WARN]  GET https://example.com/products/sku-1043 -> 429 Too Many Requests (0.3s)
[ERROR] GET https://example.com/products/sku-1044 -> 403 Forbidden (0.9s, cf-mitigated: true)
[ERROR] GET https://example.com/products/sku-1045 -> TimeoutError after 120s

Three different problems. The 429 says “slow down” — back off and retry. The 403 says “you are recognized” — retrying from the same IP is pointless. The timeout says “something is stuck” — retry, but consider whether the page needs JS rendering and a longer timeout budget. One retry strategy cannot serve all three, which is exactly why the ownership question exists.

Client-Side Retries: Writing Your Own Backoff Loop in Python

If you own retries, you own exponential backoff with jitter, attempt counting, and per-error-class handling. Here is a reasonable implementation:

import time
import random
import requests

RETRY_CONFIG = {
    "max_attempts": 5,
    "base_delay": 2.0,      # seconds
    "max_delay": 60.0,      # seconds
    "jitter": 0.5,          # fraction of delay to randomize
    "retry_statuses": {429, 500, 502, 503, 504},
}

def backoff_delay(attempt: int, cfg: dict) -> float:
    delay = min(cfg["base_delay"] * (2 ** attempt), cfg["max_delay"])
    jitter_range = delay * cfg["jitter"]
    return delay + random.uniform(-jitter_range, jitter_range)

def scrape_with_retries(url: str, cfg: dict = RETRY_CONFIG) -> requests.Response:
    last_exc = None
    for attempt in range(cfg["max_attempts"]):
        try:
            resp = requests.get(url, timeout=30)
            if resp.status_code == 403:
                # Retrying from the same IP will not help. Bail.
                raise RuntimeError(f"blocked: {url}")
            if resp.status_code in cfg["retry_statuses"]:
                last_exc = RuntimeError(f"retryable status {resp.status_code}")
            else:
                return resp
        except requests.RequestException as e:
            last_exc = e

        if attempt < cfg["max_attempts"] - 1:
            time.sleep(backoff_delay(attempt, cfg))

    raise RuntimeError(f"exhausted {cfg['max_attempts']} attempts for {url}: {last_exc}")

And the config as a dataclass, if you prefer type safety over dict access:

from dataclasses import dataclass

@dataclass
class RetryConfig:
    max_attempts: int = 5
    base_delay: float = 2.0
    max_delay: float = 60.0
    jitter: float = 0.5
    retry_statuses: frozenset = frozenset({429, 500, 502, 503, 504})

Note the explicit 403 short-circuit. That is the part most homegrown retry loops get wrong: they treat every non-2xx as “try harder,” and the backoff loop turns a soft block into a hard one. Jitter matters too — without it, a fleet of workers that started together will retry together, and you’ll synthesize your own thundering herd against the target.

This code is roughly 40 lines. It works. But it is the smallest part of the system.

The Hidden Costs of Owning the Retry Loop: Threads, Queues, and Stale Data

The loop above calls time.sleep. In a crawler, something has to absorb that sleep. If you’re thread-based, that’s a pinned thread doing nothing for up to 60 seconds. If you’re queue-based, the job sits in front of other jobs while it backs off. Either way, your infrastructure pays for the target’s bad behavior.

Here’s a local measurement from a 100-URL crawl of store.example.com, single worker, sequential:

Retry attempts per URL	Wall-clock time	Attempts issued	Notes
0	4m 12s	100	9 URLs failed outright
3	11m 40s	147	3 URLs still failed
5	19m 05s	168	2 URLs still failed

Read the last two rows carefully. Going from 3 to 5 attempts added 7+ minutes of wall-clock time and 21 extra requests, and recovered exactly one additional URL. That is the shape of the retry curve: the marginal value of attempt N decays fast, while the latency cost compounds. Attempt 2 is usually worth it. Attempt 5 rarely is.

Detecting that decay in production requires instrumentation you also have to write:

def scrape_with_metrics(url: str, cfg: dict, stats) -> requests.Response:
    for attempt in range(cfg["max_attempts"]):
        try:
            resp = requests.get(url, timeout=30)
            if resp.status_code not in cfg["retry_statuses"]:
                stats.record(url, attempt, "ok" if resp.ok else "gave_up")
                return resp
            stats.record(url, attempt, f"status_{resp.status_code}")
        except requests.RequestException as e:
            stats.record(url, attempt, type(e).__name__)

        if attempt == cfg["max_attempts"] - 1:
            stats.record(url, attempt, "exhausted")
            logger.error("retry_chain_exhausted", extra={
                "url": url, "attempts": cfg["max_attempts"],
            })
        else:
            time.sleep(backoff_delay(attempt, cfg))

The retry_chain_exhausted log line is the one you’ll alert on, because it means the target is degrading and your crawl is quietly burning money. None of this is hard. All of it is yours — the metrics schema, the alert thresholds, the dashboard, the on-call page at 2 a.m. when the failure rate doubles overnight because the target deployed a new bot check.

There’s also a data-freshness cost nobody budgets for. If your price crawler backs off 60 seconds per retry and a URL needs 4 attempts, that product’s price is 4+ minutes stale by the time you ingest it — and your pipeline treats it as “fetched now.” For most use cases that’s fine. For price or inventory monitoring during a flash sale, it isn’t.

Service-Side Retries: What a Scraping API Does Internally That You Don’t See

When you delegate the scrape, the retry loop moves into the service’s infrastructure. You submit one request; behind the curtain, the service may attempt it several times, rotating exit IPs and adjusting the request between attempts. What you get back is the final outcome, plus metadata about how it got there.

A job status response from an async scrape of store.example.com looks like this:

{
  "job_id": "job_8f3a1c",
  "status": "completed",
  "url": "https://store.example.com/products/sku-1044",
  "attempts": 3,
  "created_at": "2025-08-19T14:32:07Z"
}

(Your actual response fields will match the service’s documented schema — the point is status and the attempt count being visible to you, not hidden in a log file on a box you don’t own.)

The caller controls retry behavior through the request payload:

{
  "url": "https://store.example.com/products/sku-1044",
  "max_retries": 5,
  "auto_retry": true,
  "timeout": 120,
  "use_residential": true,
  "use_antibot": true,
  "proxy_country": "US",
  "formats": ["markdown"]
}

Two things are happening in that payload that your homegrown loop cannot replicate cheaply. First, use_residential plus internal retry means each attempt can leave from a different residential IP — a 403 on attempt 1 becomes a fresh identity on attempt 2. Second, use_antibot (enabled by default in most scraping APIs, including FineData) means each attempt carries a browser-like TLS fingerprint, so the retry isn’t just “same request, later” but “harder-to-detect request.” That’s why service-side retry success rates on blocked URLs beat same-IP client retries; it’s a mechanical advantage, not magic.

The trade-off is opacity. You don’t control the backoff curve. You can’t say “wait 90 seconds between attempts because I know this target’s rate-limit window is 60 seconds.” You get the service’s internal policy, and if it’s aggressive, you may find the retries themselves are what gets you blocked. You also add a network hop: your code calls the API, the API calls the target, so a “fast” failure now includes API overhead.

For a fuller treatment of the blocking mechanics behind those 403s, see TLS Fingerprinting Explained: How Anti-Bot Systems Detect Scrapers and Proxy Rotation Strategies for Large-Scale Web Scraping.

Per-Attempt Cost Math: Client-Side Compute vs Service-Side Pricing

Now the part that actually decides this: money. Let’s build the model with explicit assumptions, because your numbers will differ.

Assumptions for a 10,000-URL crawl of store.example.com:

• Client-side: one small worker instance at $0.04/hour, sequential-ish throughput of ~24 URLs/min, residential proxy at $5/GB with ~150KB average per page fetch (including retries), 0.5GB of proxy traffic per 100 successful fetches before retry overhead.
• Service-side: token-based pricing where a successful scrape costs 5 tokens at $0.001/token, and — critically — failed scrapes are not billed under a success-based model. Under a metered model, every attempt bills.

Approach	Cost per failed attempt	Cost per successful attempt	Total @ 10% failure	Total @ 30% failure	Total @ 60% failure
Client-side retries (own proxies)	~$0.001 compute + proxy bandwidth	~$0.001 compute + ~$0.008 proxy	~$97	~$139	~$223
Service-side, success-based billing	$0	~$0.005	~$50	~$50	~$50
Service-side, metered per attempt	~$0.005	~$0.005	~$55	~$65	~$80

The pattern to notice: client-side costs scale with failure rate, because every retry burns proxy bandwidth and worker time. Success-based service pricing is flat with respect to failure rate — the service absorbs the retry cost, which is exactly why its per-success price is higher than raw proxy cost. Metered pricing sits in between and behaves more like client-side: failures cost you.

Break-even calculation. Setting client-side total equal to success-based service total at a base of 10,000 target URLs:

• Client cost per URL ≈ $0.0008 compute/proxy for the first attempt, plus $0.001 per retry.
• Client total ≈ 10,000 × (0.0008 + 0.001 × r) where r is retries issued per URL.
• Service total ≈ 10,000 × 0.005 × (1 - f), where f is the fraction that never succeeds (you don’t pay for those under success-based billing).

At f = 0.10 and r ≈ 0.15 retries per URL (one retry on each failure): client ≈ $9.50, service ≈ $45. Client-side wins at low failure rates with cheap proxies. At f = 0.60 with r ≈ 1.2 (multiple retries chasing hard blocks): client ≈ $20 plus the cost of the 4,000 URLs you never got — which is the real number. If those 4,000 product pages were worth anything to your business, the client-side “savings” bought you an incomplete dataset.

That last point is where I think most cost comparisons go wrong. People compare infrastructure spend and forget that the output of a crawl is the product. A cheaper pipeline that returns 40% of the data is not cheaper. If you want the pricing-model mechanics broken down properly, Success-Based vs Metered Scraping API Billing Models covers it in detail.

Hybrid Pattern: Delegating Transport Retries to the Service While Keeping Business-Logic Retries Local

The correct answer for most production systems is both, with a clean split: the service retries transport failures (timeouts, 429s, 403s, connection resets) because it can change IPs and fingerprints between attempts. Your code retries application failures (empty parses, schema mismatches, truncated content) because only you know what “wrong data” looks like.

import time
import requests

API = "https://api.finedata.ai/api/v1/scrape"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}

def scrape_via_service(url: str) -> dict:
    payload = {
        "url": url,
        "formats": ["markdown"],
        "max_retries": 3,          # service retries transport failures
        "auto_retry": True,
        "timeout": 120,
        "use_residential": True,
    }
    resp = requests.post(API, json=payload, headers=HEADERS, timeout=180)
    resp.raise_for_status()
    return resp.json()

def scrape_product(url: str, local_attempts: int = 2) -> dict:
    """
    Service handles transport retries. We only retry when the
    service succeeded but the content is unusable.
    """
    for attempt in range(local_attempts):
        result = scrape_via_service(url)

        content = result.get("data", {}).get("markdown", "") or ""
        if "price" not in content.lower() or len(content) < 200:
            if attempt < local_attempts - 1:
                time.sleep(60)  # give the origin time to recover
                continue
            raise ValueError(f"content unusable after {local_attempts} attempts: {url}")
        return result

The local loop is small on purpose: two attempts, fixed 60-second delay, no exponential curve. Exponential backoff is for rate limits, and the service already handled those. What remains — a page that loaded fine but rendered an empty product template, a listing that hadn’t populated yet — is a data problem, and a slow single retry is the right medicine.

The split, in table form:

Failure category	Owner	Why
Timeout	Service	Can retry with longer budget / different route
HTTP 429	Service	Internal throttling + IP rotation beats your blind backoff
HTTP 403 / bot detection	Service	Requires fresh exit IP + fingerprint per attempt
Connection reset	Service	Transient network condition, service-side retry is cheap
HTTP 404	Neither	Not retryable; log and drop
Empty parse / missing fields	Local	Only your code knows the expected schema
Schema drift (selector broke)	Neither	Fix the extractor; retrying garbage returns garbage
Downstream write failure (DB, queue)	Local	Nothing to do with scraping; retry in your pipeline

Decision Checklist: When to Move Retries Out of Your Codebase

Score your project against these eight criteria. Whichever column wins more rows tells you where your retries belong.

#	Criterion	Favors client-side	Favors service-side
1	Failure rate on target	Under ~5%, mostly timeouts	Over ~15%, or any 403s at all
2	Team size	Solo dev, comfortable with ops	Small team, no on-call rotation for crawlers
3	Latency budget	Batch/nightly, hours available	Near-real-time freshness required
4	Proxy spend	Own residential pool already paid for	Buying residential bandwidth ad hoc
5	Target hostility	Static sites, no bot protection	JS-heavy, fingerprinting, rate limits
6	Retry observability needs	You have metrics infra and want full control	Attempt metadata in API response is enough
7	Scale	Under ~10K URLs/day	Millions of URLs/day, concurrency matters
8	Billing model tolerance	Fine paying for failures	Success-based billing available and trusted

My recommendation, stated plainly because people will disagree: if your target blocks you at all — a single 403 in your logs — stop writing retry code and delegate. Client-side retry loops are only rational against well-behaved targets with transient failures, and the moment a target deploys bot detection, every hour you spend tuning your own backoff curve is an hour spent competing with a vendor whose entire job is that one problem. I’d rather pay 5 tokens per success than employ a retry-tuning specialist.

The concrete setup I’d land on for a mid-size crawl of store.example.com:

SERVICE_PAYLOAD = {
    "url": "https://store.example.com/products/{sku}",
    "formats": ["markdown"],
    "max_retries": 3,          # service-side: transport failures
    "auto_retry": True,
    "timeout": 120,
    "use_residential": True,
    "proxy_country": "US",
}

LOCAL_RETRY = {
    "max_attempts": 2,         # local: application failures only
    "delay_seconds": 60,
}

Service-side max_retries=3 catches the transient stuff. One local retry with a 60-second delay catches the empty-template cases. Total attempts per URL cap out at 8 in the worst case, and you never re-send a blocked request from a dead IP.

Wrap-Up

Retry ownership is an architecture decision, not a coding detail. Client-side loops give you total control and cost almost nothing at low failure rates against friendly targets — but their costs scale with the target’s hostility, and they cannot fix a 403 by re-sending from the same IP. Service-side retries cost more per success but stay flat as failure rates climb, because the service can rotate identity between attempts. The hybrid pattern — transport retries delegated, application retries kept local — captures the best of both and is what I’d default to for anything running in production.

Classify your errors first. A retry strategy that can’t distinguish a 429 from a 403 from an empty parse is spending money to reproduce the same failure. If you’re also deciding between blocking and background architectures for these retries, Sync vs Async Scraping: When Blocking Requests Are Enough is the natural next read.