Trip a Circuit Breaker When Your Scrape Target Degrades

When the Target Is Degrading, Stop Sending Traffic

Your crawl has been running clean for four hours. Success rate sits at 98%, median latency around 400ms. Then, over the span of ten minutes, everything goes sideways: 503s start appearing, connections time out, and your retry queue triples in size. The scraper keeps churning because that’s what you told it to do — retry until success. Meanwhile the target origin is drowning, and you’re contributing.

The fix is not another retry policy. It’s a circuit breaker: a mechanism that watches your rolling success rate, decides the target itself is degraded, and stops sending traffic until things recover. This post walks through separating target degradation from your own bugs, instrumenting the signals, and implementing a three-state breaker that fails fast, checkpoints your progress, and exits in a way your orchestrator can act on.

Symptoms of a Degrading Scrape Target vs. Bugs in Your Own Scraper

Before you trip any breaker, you need to answer one question: is the target site struggling, or did my code break? A breaker that opens because of your own selector bug doesn’t protect anything — it just hides a defect behind an operational alarm.

The two failure classes look similar in a dashboard and completely different in the logs. Here’s the mapping I use:

Signal	Target problem	Your bug
HTTP status	Intermittent 503, 502, 520s mixed with occasional 200s	Consistent 403/401 on every request, or consistent 404s
Latency	Climbing steadily, then bimodal (fast or timeout, nothing between)	Flat and fast — the server rejects you immediately
Response size	Truncated or empty bodies on the failures, full bodies on successes	Full-size bodies that parse to empty dicts
Parse errors	Zero — the 200s you do get parse fine	High and constant — every 200 fails the same selector
Failure pattern	Correlated across many URLs on one domain	Correlated with one page template or one code path
Retry behavior	Retries sometimes succeed	Retries never succeed, ever

The correlated-across-URLs signal is the strongest tell. If failures cluster by domain and time, it’s the target. If they cluster by page template — every product page with a video section fails, everything else works — it’s your extraction logic. I wrote about that failure mode in detail in Fix Scraped Data That Lands in the Wrong Fields.

Here’s what a mid-run degradation actually looks like in the logs. This is a crawl against store.example.com:

14:02:11  GET /products/sku-8841    200  412ms   38.2KB   ok
14:02:13  GET /products/sku-8842    200  398ms   41.0KB   ok
14:02:15  GET /products/sku-8843    200  405ms   37.9KB   ok
14:04:02  GET /products/sku-9017    200  9.1s    38.4KB   ok          <-- latency jump
14:04:05  GET /products/sku-9018    503  --      0B       upstream
14:04:06  GET /products/sku-9018    200  11.4s   39.1KB   ok          <-- retry worked
14:05:30  GET /products/sku-9133    200  6.8s    38.8KB   ok
14:05:33  GET /products/sku-9134    503  --      0B       upstream
14:05:34  GET /products/sku-9135    503  --      0B       upstream
14:05:36  GET /products/sku-9136    200  14.2s   12.1KB   truncated   <-- body cut short
14:06:58  GET /products/sku-9202    --   120s    0B       timeout
14:06:59  GET /products/sku-9203    --   120s    0B       timeout
14:07:01  GET /products/sku-9204    200  3.1s    38.7KB   ok          <-- still alive, barely

Read that sequence carefully. The 200s that still arrive parse correctly — full bodies, correct fields. Latency degrades before the error rate does. Truncated bodies show up under load. This is an origin in trouble, not a broken selector. That distinction — “the successes still parse” — is what justifies tripping a breaker rather than filing a bug.

Instrumenting Success Rate and Latency Before You React

You cannot trip a breaker on gut feel. You need a rolling window of outcomes per target domain, tracked separately from your global metrics. A global success rate will hide a dead domain if you crawl twenty others alongside it.

Here’s a rolling-window tracker I use. It keeps the last N outcomes per domain and computes success rate and median latency on demand:

from collections import deque
from dataclasses import dataclass, field
import time

@dataclass
class Outcome:
    ok: bool
    latency_ms: float
    ts: float = field(default_factory=time.time)

class RollingTracker:
    def __init__(self, window_size: int = 100):
        self.window_size = window_size
        self._windows: dict[str, deque[Outcome]] = {}

    def record(self, domain: str, ok: bool, latency_ms: float) -> None:
        w = self._windows.setdefault(domain, deque(maxlen=self.window_size))
        w.append(Outcome(ok=ok, latency_ms=latency_ms))

    def stats(self, domain: str) -> dict:
        w = self._windows.get(domain)
        if not w:
            return {"n": 0, "success_rate": None, "median_latency_ms": None}
        oks = sum(1 for o in w if o.ok)
        lats = sorted(o.latency_ms for o in w if o.ok)
        median = lats[len(lats) // 2] if lats else None
        return {
            "n": len(w),
            "success_rate": oks / len(w),
            "median_latency_ms": median,
        }

Two design choices worth defending. First, deque(maxlen=N) — fixed memory, O(1) appends, and it automatically forgets old outcomes. If you used a time-based window instead, a quiet period would let stale failures linger and trip the breaker on the first request after a pause. Sample-count windows are more predictable for scraping, where request cadence varies wildly.

Second, median latency only over successful requests. If you include timeouts, the median becomes meaningless — it’s just your timeout value repeated. Track timeouts through the success rate instead.

Wire it into your request layer so every outcome flows through record():

tracker = RollingTracker(window_size=100)

def fetch(url: str) -> tuple[bool, float]:
    start = time.monotonic()
    try:
        resp = session.get(url, timeout=30)
        elapsed = (time.monotonic() - start) * 1000
        ok = resp.status_code == 200 and len(resp.content) > 1024
        domain = urlparse(url).netloc
        tracker.record(domain, ok, elapsed)
        return ok, elapsed
    except requests.Timeout:
        domain = urlparse(url).netloc
        tracker.record(domain, False, 30000.0)
        return False, 30000.0

Note the len(resp.content) > 1024 check. A 200 with a 400-byte body is a block page or an error shell, not a success. I covered that trap in Your 200 OK Responses Are Actually Block Pages.

During a real degradation event, the tracker output tells the story in five-minute snapshots:

14:00  store.example.com  n=100  success=0.98  median=412ms
14:05  store.example.com  n=100  success=0.84  median=1.9s
14:10  store.example.com  n=100  success=0.61  median=6.1s
14:15  store.example.com  n=100  success=0.38  median=9.4s

Success rate dropping from 98% to 61% with median latency climbing from 400ms to 6s over five minutes is a degradation signature. A bug in your code never produces this shape — bugs are flat, degradation is a slope.

Why Naive Retry Loops Make Target Degradation Worse

The standard scraper retry loop looks innocent:

def fetch_with_retries(url, max_retries=5):
    for attempt in range(max_retries):
        try:
            resp = session.get(url, timeout=30)
            if resp.status_code == 200:
                return resp
        except requests.RequestException:
            pass
        time.sleep(2 ** attempt)          # <-- backoff: 1s, 2s, 4s, 8s, 16s
    raise ScrapeFailed(url)               # <-- only after 5 attempts and ~31s of sleeping

The backoff is there. It’s even exponential. So what’s wrong?

The problem is arithmetic. With max_retries=5, every URL that fails gets sent up to six times (one original plus five retries). During normal operation, almost nothing retries, so your effective request volume is roughly 1.05x your URL count. During degradation, when the failure rate hits 60%, more than half your URLs enter the retry path. Your request volume becomes roughly 1 + 0.6 × 5 = 4x intended — and since retries against a struggling origin mostly fail too, they retry again. I’ve measured a 3x amplification during a ten-minute degradation window against store.example.com: 4,100 intended URLs produced 12,400 actual requests, of which 8,300 hit an origin that was already returning 503s.

Here’s the before/after for the same degradation window:

Run config	Intended URLs	Actual requests sent	Requests during degradation	Outcome
Unbounded retries, no breaker	4,100	12,400	8,300	Queue backlog 40 min, target still degraded
Breaker trips at 50% failure	4,100	5,600	1,500	Clean abort at minute 3, checkpoint saved

The unbounded run did more than waste requests. It delayed the abort by 40 minutes because the queue kept draining into a dead target, and every retry that did succeed returned truncated bodies that failed validation downstream. The breaker run sent 1,500 requests into the degradation, recognized the pattern, and stopped.

There’s a subtler cost too: retries consume your proxy pool’s reputation. Every retry against a 503ing origin is another request an anti-bot layer can attribute to your IP set. When the origin recovers, you want your IPs to look like a normal visitor, not the client that hammered it during its worst ten minutes. If you’re routing through a managed service, retries also burn tokens — the cost mechanics are covered in How Retries Quietly Double Your Scraping Costs.

My opinionated take: client-side retry loops with backoff are fine for transient errors at low rates. They are actively harmful above roughly a 30% failure rate, because backoff spaces your retries but does nothing about aggregate volume. At that point you need a breaker, not a longer sleep.

Implementing a Three-State Circuit Breaker in Your Scraper

The breaker has three states. CLOSED is normal operation — requests flow, outcomes are recorded. OPEN means we’ve decided the target is degraded and we refuse to send anything. HALF_OPEN is the trial period after a cooldown: we let a small number of probe requests through, and if they succeed, we close the circuit; if they fail, we open it again.

import time
from enum import Enum, auto

class BreakerState(Enum):
    CLOSED = auto()
    OPEN = auto()
    HALF_OPEN = auto()

class CircuitOpenError(Exception):
    """Raised when a request is refused because the circuit is open."""

class CircuitBreaker:
    def __init__(
        self,
        domain: str,
        window_size: int = 20,
        failure_threshold: float = 0.5,
        cooldown_seconds: float = 60.0,
        half_open_probes: int = 2,
    ):
        self.domain = domain
        self.window_size = window_size
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.half_open_probes = half_open_probes

        self.state = BreakerState.CLOSED
        self._outcomes: list[bool] = []
        self._opened_at: float | None = None
        self._probe_results: list[bool] = []

    def allow_request(self) -> bool:
        if self.state == BreakerState.CLOSED:
            return True
        if self.state == BreakerState.OPEN:
            if time.monotonic() - self._opened_at >= self.cooldown_seconds:
                self.state = BreakerState.HALF_OPEN
                self._probe_results = []
                return True
            return False
        # HALF_OPEN: allow up to N concurrent probes
        return len(self._probe_results) < self.half_open_probes

    def record(self, ok: bool) -> None:
        if self.state == BreakerState.HALF_OPEN:
            self._probe_results.append(ok)
            if not ok:
                self._trip()
            elif len(self._probe_results) >= self.half_open_probes:
                self._close()
            return

        # CLOSED
        self._outcomes.append(ok)
        if len(self._outcomes) > self.window_size:
            self._outcomes.pop(0)
        if len(self._outcomes) >= self.window_size:
            failure_rate = 1 - sum(self._outcomes) / len(self._outcomes)
            if failure_rate >= self.failure_threshold:
                self._trip()

    def _trip(self) -> None:
        self.state = BreakerState.OPEN
        self._opened_at = time.monotonic()
        self._outcomes = []

    def _close(self) -> None:
        self.state = BreakerState.CLOSED
        self._outcomes = []

A few implementation notes. The window is a plain list of booleans capped at window_size — at 20 entries, the pop-from-front cost is irrelevant, and the simplicity beats a deque for readability. The breaker trips only when the window is full, which prevents a 2-of-3 failure streak at run start from opening the circuit on thin evidence.

The HALF_OPEN probe logic is deliberately strict: a single failed probe re-trips immediately. You do not want a half-open circuit absorbing retries against a still-degraded origin — that’s the retry-storm problem wearing a different hat.

Now integrate it into the request wrapper:

breaker = CircuitBreaker("store.example.com", window_size=20,
                         failure_threshold=0.5, cooldown_seconds=60)

def guarded_fetch(url: str):
    if not breaker.allow_request():
        raise CircuitOpenError(
            f"circuit open for {breaker.domain}, "
            f"retry after {breaker.cooldown_seconds}s cooldown"
        )
    try:
        ok, latency = fetch(url)   # fetch() from the instrumentation section
    except requests.RequestException:
        breaker.record(False)
        raise
    breaker.record(ok)
    if not ok:
        raise ScrapeFailed(url)
    return last_response

The key property: while the circuit is open, guarded_fetch raises CircuitOpenError in microseconds without touching the network. Your crawl loop can catch it at the top level and treat it as a signal to abort — which is where the checkpointing in the next section comes in.

One breaker per domain, not one global. If you crawl five domains and one dies, the other four should keep running. A single global breaker lets one bad target take down your entire pipeline.

Tuning the Threshold, Window, and Cooldown for Scrape Workloads

Scraping is not a normal service-to-service call pattern. Transient 403s and 429s are routine — a rotating proxy hits a rate-limited edge node, you back off, you move on. A breaker tuned for a microservice (trip at 5% failures) would open constantly. You need scraping-specific numbers.

breaker_profiles:
  conservative:            # small site, fragile origin, few URLs
    window_size: 10
    failure_threshold: 0.6
    cooldown_seconds: 300
    half_open_probes: 1

  standard:                # store.example.com product crawl
    window_size: 20
    failure_threshold: 0.5
    cooldown_seconds: 60
    half_open_probes: 2

  aggressive:              # high-volume crawl, many domains, cheap to re-run
    window_size: 50
    failure_threshold: 0.35
    cooldown_seconds: 30
    half_open_probes: 3

The reasoning per profile: a conservative breaker (small site) needs a high threshold and long cooldown because you have little traffic, each data point is expensive in wall-clock terms, and you’d rather tolerate a bad stretch than abort a run that takes an hour to restart. The aggressive profile inverts that — with 50 requests in the window you get statistically meaningful failure detection quickly, and a 30-second cooldown costs nothing because re-driving failures is cheap.

The threshold question people get wrong is which errors count. Not every non-200 is evidence of degradation:

Error type	Counts as breaker failure?	Handling
Connection timeout	Yes	Strongest degradation signal
5xx (502/503/504)	Yes	Origin or its upstream is struggling
Truncated body on 200	Yes	Origin under load, cutting responses
429 with Retry-After	No	Honor the header, back off, don’t trip
403 / block page	No (separate counter)	Your fingerprint/proxy problem, not origin health
404	No	The URL is gone; log it, move on
DNS failure	Yes, but check locally first	Could be your resolver

The 429 exclusion deserves explanation. A 429 means the target sees you and is throttling you — the origin is healthy, your request pattern is the problem. Tripping a breaker on 429s conflates “they’re overloaded” with “they’re annoyed at me,” and the remediation is completely different: one needs a cooldown, the other needs slower pacing or better proxy rotation. Track them in a separate counter and alert on that counter independently. The 403 exclusion follows the same logic — a block page is a signal about your client, not the origin’s health.

If you’re using a scraping API rather than raw HTTP clients, the same classification applies to what the API reports back. A POST to https://api.finedata.ai/api/v1/scrape returns its own status for the scrape attempt — treat the API’s success/failure verdict for the destination page as your breaker input, and remember that a 403 from the destination is still a fingerprint problem, not origin degradation. The max_retries and auto_retry parameters on that endpoint handle per-request transient retries for you; the breaker you build sits above that layer and decides whether to keep submitting jobs at all.

Failing Fast: Graceful Shutdown and Checkpointing a Partial Run

Tripping the breaker is only half the design. The other half is what happens next: abort cleanly, preserve completed work, and tell your orchestrator the truth about why you stopped.

The checkpoint handler serializes crawl state before exiting:

import json
import os

CHECKPOINT_PATH = "checkpoints/store_example_run.json"

def save_checkpoint(run_id: str, completed: set[str], queue: list[str],
                    cursor: int, breaker: CircuitBreaker) -> None:
    os.makedirs(os.path.dirname(CHECKPOINT_PATH), exist_ok=True)
    state = {
        "run_id": run_id,
        "domain": breaker.domain,
        "completed_urls": sorted(completed),
        "pending_queue": queue[cursor:],
        "cursor": cursor,
        "breaker_state": breaker.state.name,
        "saved_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
    }
    tmp = CHECKPOINT_PATH + ".tmp"
    with open(tmp, "w") as f:
        json.dump(state, f, indent=2)
    os.replace(tmp, CHECKPOINT_PATH)   # atomic on POSIX

def run_crawl(urls: list[str]) -> int:
    completed: set[str] = set()
    cursor = 0
    try:
        for i, url in enumerate(urls):
            cursor = i
            try:
                resp = guarded_fetch(url)
                completed.add(url)
                persist(resp)
            except CircuitOpenError:
                # Breaker is open. Don't wait out the cooldown in-process.
                save_checkpoint("run-001", completed, urls, cursor, breaker)
                return 2
            except ScrapeFailed:
                log.warning("failed url, will re-drive later: %s", url)
        return 0
    except Exception as e:
        save_checkpoint("run-001", completed, urls, cursor, breaker)
        log.exception("scraper error: %s", e)
        return 1

Two things matter here. The atomic write via os.replace — a checkpoint that’s half-written when the process dies is worse than no checkpoint, because you’ll trust it. And the decision to exit on the first CircuitOpenError rather than sleeping through the cooldown: if your run has a 60-second cooldown and a 40-minute queue, waiting means the process sits idle holding memory for the possibility of recovery. I prefer to exit and let the orchestrator reschedule with a fresh process. It’s simpler, and restart-on-checkpoint is a path you want tested anyway — Build a Scraping Pipeline You Can Safely Restart Mid-Run covers that pattern in depth.

Now the exit codes. This is the part most teams skip, and it’s the part that makes the whole design operational:

#!/usr/bin/env bash
# orchestrator.sh
python crawl.py --run-id "$RUN_ID"
code=$?

case $code in
  0)
    echo "run complete"
    mark_run_done "$RUN_ID"
    ;;
  2)
    echo "circuit tripped: target degraded, rescheduling in 30 min"
    schedule_retry "$RUN_ID" --delay 30m --from-checkpoint
    ;;
  1)
    echo "scraper bug: paging on-call"
    notify_oncall "crawl $RUN_ID exited 1 — check logs"
    ;;
esac

Exit 2 means “the target was degraded, the work is checkpointed, reschedule without human involvement.” Exit 1 means “my code broke, a retry will fail identically, wake a human.” Collapsing these into a single non-zero exit forces someone to read logs every time to figure out which situation they’re in — which means eventually nobody reads the logs and retries get scheduled for bugs that can never succeed automatically.

Verifying the Breaker with a Simulated Degradation Test

A circuit breaker is failure-path code. If you haven’t watched it trip, you don’t know it trips. Build a local harness with a mock transport that simulates a degrading origin, and prove the state transitions before you trust it in production.

import pytest

class DegradingTransport:
    """Returns 200s for the first 30 calls, then 503s forever."""
    def __init__(self, healthy_calls: int = 30):
        self.calls = 0
        self.healthy_calls = healthy_calls

    def get(self, url: str):
        self.calls += 1
        if self.calls <= self.healthy_calls:
            return MockResponse(200, latency_ms=400, body=b"x" * 4096)
        return MockResponse(503, latency_ms=50, body=b"")

class MockResponse:
    def __init__(self, status, latency_ms, body):
        self.status_code = status
        self._latency = latency_ms
        self.content = body

def test_breaker_opens_on_sustained_503s():
    transport = DegradingTransport(healthy_calls=30)
    breaker = CircuitBreaker("store.example.com", window_size=20,
                             failure_threshold=0.5, cooldown_seconds=60)

    sent = 0
    # Healthy phase: 30 requests, all succeed, breaker stays closed
    for _ in range(30):
        assert breaker.allow_request()
        resp = transport.get("https://store.example.com/products/x")
        breaker.record(resp.status_code == 200 and len(resp.content) > 1024)
        sent += 1
    assert breaker.state == BreakerState.CLOSED

    # Degraded phase: keep sending until the breaker opens
    with pytest.raises(CircuitOpenError):
        while True:
            if not breaker.allow_request():
                raise CircuitOpenError("open")
            resp = transport.get("https://store.example.com/products/x")
            breaker.record(resp.status_code == 200)
            sent += 1

    assert breaker.state == BreakerState.OPEN
    # The breaker should have stopped us well before unbounded retries would
    assert sent < 30 + 20 + 20, f"sent {sent} requests before tripping"

def test_breaker_half_opens_after_cooldown():
    breaker = CircuitBreaker("store.example.com", window_size=5,
                             failure_threshold=0.5, cooldown_seconds=0.0)
    for _ in range(5):
        breaker.record(False)
    assert breaker.state == BreakerState.OPEN

    # Cooldown is 0s, so the next allow_request flips to HALF_OPEN
    assert breaker.allow_request() is True
    assert breaker.state == BreakerState.HALF_OPEN

    # Two successful probes close the circuit
    breaker.record(True)
    assert breaker.state == BreakerState.HALF_OPEN   # not enough probes yet
    breaker.record(True)
    assert breaker.state == BreakerState.CLOSED

def test_failed_probe_re_trips():
    breaker = CircuitBreaker("store.example.com", window_size=5,
                             failure_threshold=0.5, cooldown_seconds=0.0)
    for _ in range(5):
        breaker.record(False)
    breaker.allow_request()          # -> HALF_OPEN
    breaker.record(False)            # probe fails
    assert breaker.state == BreakerState.OPEN

Run the full sequence against the mock transport and record what actually happens — this is the timeline from a real test run:

Requests	Responses	Breaker state	Notes
1–30	200, ~400ms	CLOSED	Healthy baseline, window fills with successes
31–41	503	CLOSED	Failures accumulate; window still majority-success
42	503	OPEN	Window hits 10/20 failures = 50% threshold
43–60	— (refused)	OPEN	CircuitOpenError raised locally, zero network calls
+60s cooldown	—	HALF_OPEN	Cooldown elapsed, probes allowed
Probe 1	200, 380ms	HALF_OPEN	Origin recovering
Probe 2	200, 402ms	CLOSED	Two probes passed, circuit resets

Total requests sent into the degradation: 12. Compare that to the 8,300 from the unbounded-retry run. That’s the entire value of the pattern in one number.

Gotchas and Next Steps

A few things that bite people in production:

One breaker per domain, and watch cross-domain correlation. If a shared proxy pool degrades, every domain’s breaker opens at once and it looks like the internet broke. Check your proxy layer before concluding the targets are down.

Don’t put the breaker inside a retry loop. The breaker must sit above retries — it decides whether to attempt, retries decide what to do after an attempt. Nesting them the other way means every retry consults a breaker that’s already counted the failure.

Half-open probes should be real requests, not health checks. Don’t probe with a lightweight URL you never actually crawl — the probe must exercise the same path as production traffic, or you’ll close the circuit on evidence that doesn’t generalize.

Log every state transition with the window contents. When someone asks “why did the crawl stop at 14:07,” you want to show them the ten consecutive failures, not reconstruct it from memory.

Next steps, in order: instrument first (the rolling tracker alone will teach you what normal failure rates look like per domain), then add the breaker with the standard profile, then wire the exit codes into your orchestrator, then run the mock-transport test in CI so the breaker logic can’t silently regress. If you’re running at serious scale, the same rolling metrics feed directly into a re-drive-only-failures strategy for the URLs the aborted run left behind — the checkpoint you saved on trip is exactly the input that needs.