Your 200 OK Responses Are Actually Block Pages

Why a 200 OK Is the Least Informative Part of an HTTP Response

Your scraper logged 48,000 fetches last night. Every single one returned 200. Your success-rate dashboard shows 100%. And this morning, your product database is full of CAPTCHA pages.

That’s the actual symptom, and it’s more common than transport-level failure. Anti-bot systems learned years ago that operators monitor status codes, so the cheapest countermeasure available to them is to serve the block with a 200. Nothing at the HTTP layer says “this response is a lie.” The status code describes whether the server sent bytes back, not whether those bytes are the page you asked for.

Here are three real-shaped responses from store.example.com, all fetched within the same hour, all with the same status:

	Product page	CAPTCHA interstitial	Rate-limit block
Status	200	200	200
Content-Type	text/html	text/html	text/html
Body length	214 KB	6.1 KB	4.3 KB
Title	”Ergonomic Desk Lamp – Store Example"	"Just a moment…"	"Access denied”
cf-mitigated header	absent	challenge	absent
set-cookie	session cookie	challenge cookie + clearance token	none
Product data in DOM	yes	no	no

Three completely different outcomes. One status code. If your pipeline branches on response.status_code == 200, all three of these land in the same bucket and the third one poisons your dataset silently.

You can sometimes catch this before reading the body, if you know which headers to read. Here’s an annotated response from example.com:

HTTP/1.1 200 OK
content-type: text/html; charset=UTF-8
cf-mitigated: challenge              ← challenge marker; a real page never carries this
set-cookie: __cf_bm=abc123...;        ← bot-management cookie, fine on its own,
      Path=/; Max-Age=1800               but paired with cf-mitigated it's a
                                         challenge handshake
set-cookie: cf_clearance=...;          ← clearance token only appears when a
      Path=/                              challenge was served or solved
x-frame-options: SAMEORIGIN           ← weak signal; common on real pages too
server: cloudflare                    ← tells you which playbook to expect
cache-control: no-store               ← challenges are never cached; real pages
                                         often are

Two caveats before you go build a header-based detector. First, header names differ per vendor, and none of them are a documented contract — treat them as hints, not verdicts. Second, plenty of challenge pages carry no distinguishing headers at all. The body is where the truth lives. We wrote a whole separate piece on how anti-bot systems fingerprint your client at the TLS layer — this article is about the other direction: detecting what they send back to you.

The Four Kinds of Fake Success: Challenge, Interstitial, Empty Shell, and Soft Block

“Block page” is a lazy category. There are at least four distinct things hiding inside a 200, and they need different responses. Here’s each one as it actually appears in the HTML.

1. The JS challenge page — a meta refresh or script-driven reload loop that a plain HTTP client never gets past:

<!-- store.example.com — JS challenge -->
<html>
<head>
  <title>Checking your browser...</title>
  <meta http-equiv="refresh" content="3; url=/?__cf_chl_jschl_tk__=...">
  <script>
    (function(){var s,t,o,p,b,r,e,a,k,i,n,g,f,...=+"...";
    // obfuscated blob, evaluates and submits a form
  })();
</script>
</head>
<body>Redirecting... please wait.</body>
</html>

2. The CAPTCHA form — the page loaded, but it’s a form, not content:

<!-- store.example.com — CAPTCHA interstitial -->
<html>
<head><title>Verify you are human</title></head>
<body>
  <form id="challenge-form" action="/___captcha_verify" method="POST">
    <div class="g-recaptcha" data-sitekey="..."></div>
    <input type="submit" value="Submit">
  </form>
</body>
</html>

3. The empty shell — the nastiest one. The template renders, navigation and footer and all, but the data region is empty. Often happens when the server decides to degrade rather than block:

<!-- store.example.com — category page, zero listings -->
<html>
<head><title>Desk Lamps – Store Example</title></head>
<body>
  <nav>...full navigation, 40 links...</nav>
  <div class="product-grid"><!-- empty --></div>
  <footer>...full footer...</footer>
</body>
</html>

4. The soft block — the site template with a “no results” or “unavailable in your region” message where the data should be. Status 200, valid markup, plausible-looking at a glance:

<!-- store.example.com — soft block / no results -->
<html>
<head><title>Desk Lamps – Store Example</title></head>
<body>
  <nav>...full navigation...</nav>
  <div class="product-grid">
    <p class="no-results">No products found. Try a different search.</p>
  </div>
</body>
</html>

Map each mode to its detection signal:

Failure mode	Title tag	Body length	<form> present	Data-region element count	Correct response
JS challenge	”Checking your browser…“	tiny (<10 KB)	maybe	0	escalate to render tier
CAPTCHA form	”Verify you are human”	tiny	yes, with captcha class	0	backoff, change exit IP
Empty shell	correct title	normal	no	0	re-fetch, flag pattern
Soft block	correct title	normal	no	0 or 1 (no-results)	treat as failure, do not retry blindly

The empty shell and soft block are why body-size heuristics alone aren’t enough. Their byte counts look identical to a healthy page. You need content-aware checks, which we’ll get to. But start cheap, because cheap checks run on every fetch.

Cheap Signals First: Title Tags, Body Size, and Response Fingerprints

Most challenge pages are generated from a handful of templates. That means they’re byte-identical across thousands of requests — which makes them trivially detectable with a hash. This runs in microseconds, before any parser touches the body.

import hashlib
import re
from dataclasses import dataclass, field

KNOWN_CHALLENGE_TITLES = {
    "just a moment...",
    "checking your browser...",
    "verify you are human",
    "access denied",
    "attention required",
    "security check",
}

TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)

@dataclass
class FetchVerdict:
    body_hash: str = ""
    title: str = ""
    title_match: bool = False          # title matches known challenge list
    size_anomaly: bool = False         # body far below domain's normal band
    suspicious: bool = field(init=False)

    def __post_init__(self):
        self.suspicious = self.title_match or self.size_anomaly

def fingerprint(body: bytes, normal_min_bytes: int = 15_000) -> FetchVerdict:
    head = body[:512]                  # challenge templates differ at the top
    body_hash = hashlib.sha256(head).hexdigest()
    m = TITLE_RE.search(body[:4096].decode("utf-8", errors="ignore"))
    title = (m.group(1).strip().lower() if m else "")
    return FetchVerdict(
        body_hash=body_hash,
        title=title,
        title_match=title in KNOWN_CHALLENGE_TITLES,
        size_anomaly=len(body) < normal_min_bytes,
    )

Two design decisions worth defending. Hashing only the first 512 bytes keeps the fingerprint stable even when the challenge template embeds a per-request token further down the body — the boilerplate at the top doesn’t change. And the title regex reads from the first 4 KB rather than parsing the whole document, because challenge pages put <title> near the top and you want this check to cost almost nothing.

The normal_min_bytes threshold shouldn’t be guessed. Measure it. Run a small experiment: fetch the same URL on store.example.com a hundred times over a day, log body lengths, and look at the distribution. In a typical run you’ll see something like this:

$ python measure_sizes.py --url https://store.example.com/desk-lamps --n 100

body length distribution (bytes):
  real pages:     min 198k   p50 214k   max 231k   count 91
  challenge pages: min 5.9k  p50 6.1k   max 6.4k   count 9

  gap between clusters: >190k

Two clusters, an enormous gap between them, no overlap. That’s the common case, and it’s why a size threshold works so well for challenge detection. Set normal_min_bytes somewhere in the empty middle — 15 KB here is conservative. If your target page is naturally small (a JSON API response, a lightweight status page), size heuristics are useless and you skip straight to structural checks. Don’t cargo-cult the threshold; measure per domain.

Structural Detection: Counting the DOM Elements That Should Exist

Byte-level heuristics catch challenges. They don’t catch empty shells, because a shell page has normal size, a correct title, and a fully rendered template. The only reliable signal for a shell is: the elements that carry the data aren’t there.

Define a content contract per page type. A product page on store.example.com must contain a price, an add-to-cart control, and a title. If any of those is missing, the fetch failed — regardless of status code.

from selectolax.parser import HTMLParser

# The contract: what MUST exist for this fetch to count as success.
PRODUCT_CONTRACT = {
    ".product-title": 1,
    ".price": 1,
    ".add-to-cart": 1,
    ".product-specs li": 5,   # at least 5 spec rows
}

def check_contract(html: str, contract: dict[str, int]) -> tuple[bool, dict]:
    tree = HTMLParser(html)
    report = {}
    passed = True
    for selector, minimum in contract.items():
        count = len(tree.css(selector))
        ok = count >= minimum
        report[selector] = {"found": count, "required": minimum, "ok": ok}
        passed = passed and ok
    return passed, report

The minimum values matter more than they look like they should. ".price": 1 catches the empty shell. ".product-specs li": 5 catches something worse: a partially degraded page where the server renders the price but drops the spec table because an upstream service is rate-limiting it. Partial data is the failure mode that survives naive checks and quietly corrupts downstream analytics. If you assert only “page has a price,” you’ll ship records with prices and missing specs for weeks before anyone notices.

Selectors rot, though. A redesign breaks .add-to-cart and your contract starts failing every legitimate page. Two mitigations: keep contracts in config, not code, and give each selector a fallback chain:

PRICE_SELECTORS = [".price", "[data-testid='price']", ".product-info .cost"]

def count_with_fallback(tree, selectors: list[str], minimum: int) -> int:
    for sel in selectors:
        n = len(tree.css(sel))
        if n >= minimum:
            return n
    return 0

When the primary selector fails but a fallback passes, log it. A fallback firing is your early warning that the site changed — wire that into the same alerting we cover below. We’ve written a longer playbook for when a redesign breaks your extraction; the contract file is the tripwire that tells you it happened.

Here’s the per-domain config I’d actually ship:

# contracts/store.example.com.yaml
domain: store.example.com
page_types:
  product:
    url_pattern: "/p/{slug}"
    min_body_bytes: 150000
    required:
      - selectors: [".product-title", "h1.product-name"]
        min: 1
      - selectors: [".price", "[data-testid='price']"]
        min: 1
      - selectors: [".add-to-cart", "button[data-action='add']"]
        min: 1
      - selectors: [".product-specs li"]
        min: 5
    forbidden:                       # presence of these = block, not content
      - ".no-results"
      - "#challenge-form"
      - ".g-recaptcha"
  category:
    url_pattern: "/c/{slug}"
    min_body_bytes: 100000
    required:
      - selectors: [".product-grid .product-card"]
        min: 12                      # a real category page has 24-48 cards
    forbidden:
      - ".no-results"

Note the forbidden list — it’s the inverse contract. .no-results present means soft block even if the title and size look fine. And note min: 12 on product cards: a category page with 2 cards out of 24 is a degraded response, not a success. Set thresholds from observed healthy pages, with slack.

Handling JavaScript-Only Challenges Without Rendering Every Page

The obvious fix for JS challenges is to render everything with a headless browser. That’s also the expensive fix — rendering costs 5-10x the time and tokens of a plain fetch, and most of your pages don’t need it. On a nightly run of 50,000 URLs, rendering everything to catch the 3% that got challenged is paying a 100% premium for a 3% problem.

The right architecture is two-tier. Tier one: plain fetch, cheap detection. Tier two: render only what tier one flags.

Detection happens on the raw HTML before any rendering decision:

import re

CHALLENGE_MARKERS = [
    re.compile(rb"__cf_chl_", re.I),            # challenge token param
    re.compile(rb"challenge-form", re.I),
    re.compile(rb"cf-chl-bypass", re.I),
    re.compile(rb"checking your browser", re.I),
    re.compile(rb"verify you are human", re.I),
    re.compile(rb"<noscript>[^<]{0,400}enable javascript", re.I | re.S),
    re.compile(rb"onload=[\"']?window\.location", re.I),  # instant JS redirect
]

# A long obfuscated inline blob is itself a strong signal
OBFUSCATED_BLOB = re.compile(
    rb"<script>\s*\(function\(\)\{var [a-z,]{20,}=", re.I
)

def needs_render(body: bytes) -> bool:
    if any(p.search(body) for p in CHALLENGE_MARKERS):
        return True
    return bool(OBFUSCATED_BLOB.search(body[:8192]))

Flagged URLs go to an escalation queue, not back into the main fetch loop:

# escalation.yaml
tiers:
  fetch:
    engine: plain_http
    detect: [fingerprint, contract]
    on_flag: escalate
  render:
    engine: api_scrape          # POST /api/v1/scrape
    options:
      use_js_render: true       # headless browser, +5 tokens
      use_antibot: true         # browser-like TLS, on by default
      max_retries: 3
      timeout: 120
      js_wait_for: "networkidle"
    revalidate: contract        # same content contract, no special-casing
    on_fail: quarantine

The critical line is revalidate: contract. Escalated fetches get validated against the exact same content contract as tier-one fetches. If you skip this, you’ll eventually “pass” a rendered challenge page into your dataset because the render tier succeeded at loading a page — just the wrong page. Escalation changes the engine, not the definition of success.

If you’re doing this against a scraping API rather than your own Playwright fleet, the same two-tier shape applies: submit the cheap fetch first, and only the flagged re-fetch carries use_js_render: true. The token math is the same either way — you pay the render premium on the challenged minority, not the whole run. Whether you run your own browser fleet or rent one is a separate trade-off we’ve compared in JS rendering versus plain HTTP cost.

Instrumenting the Pipeline: Log the Signal, Not Just the Status

Detection code is worthless if its output dies in a log line nobody reads. The failure story at the top of this article — 48,000 “successes” that were mostly blocks — happens because pipelines log status_code and nothing else. Restructure the log record so every fetch carries its verdict:

{
  "ts": "2026-01-14T02:31:07Z",
  "domain": "store.example.com",
  "url": "https://store.example.com/p/desk-lamp-04",
  "http_status": 200,
  "body_bytes": 6122,
  "body_hash": "9f2c...",
  "title": "just a moment...",
  "title_match": true,
  "size_anomaly": true,
  "contract_pass": false,
  "contract_report": {
    ".product-title": {"found": 0, "required": 1},
    ".price": {"found": 0, "required": 1}
  },
  "failure_mode": "js_challenge",
  "escalation_path": "fetch -> render",
  "escalated": true,
  "final_verdict": "retry_scheduled",
  "latency_ms": 412
}

Every field earns its place. body_hash lets you group identical challenge templates and count distinct block variants. failure_mode gives you a categorical metric instead of a boolean. escalation_path tells you what the pipeline did about it, and final_verdict closes the loop — a record ending in retry_scheduled that never resolves is a stuck job, which is its own alert.

Then wire the categorical signal into alerting. A spike in challenge detections should page someone before the data quality degrades:

from collections import Counter
import time

class ChallengeRateAlert:
    def __init__(self, window_sec=600, threshold=0.05):
        self.window = window_sec
        self.threshold = threshold
        self.events = []  # (timestamp, domain, failure_mode)

    def record(self, domain: str, failure_mode: str | None):
        now = time.time()
        self.events.append((now, domain, failure_mode))
        self.events = [e for e in self.events if now - e[0] < self.window]

    def check(self, domain: str) -> bool:
        domain_events = [e for e in self.events if e[1] == domain]
        if len(domain_events) < 20:      # too little data to judge
            return False
        blocked = sum(1 for e in domain_events if e[2])
        rate = blocked / len(domain_events)
        return rate > self.threshold     # >5% challenges in 10 min = alert

The minimum-sample guard (< 20) matters. At 3 AM your crawler might fetch two URLs in ten minutes, both challenged — a 100% rate that’s statistically meaningless. Alert on sustained rates, not single events.

One opinion here that people push back on: alert on detection rate, not on success rate. Your success-rate metric is downstream of detection — if detection breaks, success rate lies to you confidently. The detection signal is the primitive; build the dashboard on it.

Responding to Detection: Backoff, Retry, and Cache-Quarantine Strategies

Detection without a response plan is just faster disappointment. What you do next depends on which failure mode you classified, and the classification work in the earlier sections pays off exactly here.

The core loop, with backoff, jitter, and a domain-level circuit breaker:

ON challenge_detected(url, domain, mode):

    IF mode == "js_challenge":
        enqueue(url, tier="render", delay=0)
        # challenges are often per-session; render tier usually clears them

    ELSE IF mode in ("captcha", "rate_limit_block"):
        attempt = attempts.get(url, 0)
        IF attempt >= 3:
            quarantine(url, reason=mode)
            RETURN

        base = 30 * (2 ** attempt)          # 30s, 60s, 120s
        delay = base + uniform(0, base * 0.5)   # jitter: 0-50% of base
        schedule_retry(url, delay=delay, rotate_exit_ip=True)
        increment(domain_counter[domain], window=600s)

    ELSE IF mode in ("empty_shell", "soft_block"):
        # NOT transient. Retrying the same way will reproduce it.
        quarantine(url, reason=mode, review="check exit IP reputation / geo")

    # Circuit breaker: sustained blocks mean stop, not slow down
    IF domain_block_rate(domain, 600s) > 0.30:
        pause_domain(domain, duration=1800s)
        alert("circuit_open", domain)

The jitter is not optional decoration. If you’re running 40 parallel workers and they all hit a challenge at the same moment, synchronized exponential backoff turns your retry storm into a synchronized retry storm. Jitter de-correlates them.

The circuit breaker threshold — 30% blocks in ten minutes — is deliberately aggressive. When a domain is actively blocking you, additional requests burn proxies, tokens, and whatever reputation your exit IPs had left. Pausing for 30 minutes costs you latency on one domain. Not pausing can cost you the ability to scrape that domain at all for days.

Which response fits which failure mode:

Response	How it works	Right for	Wrong for
Immediate retry	re-fetch now, same config	transient blips, single-page glitch	anything sustained; doubles your request rate against a domain that’s already annoyed
Backoff + retry	exponential delay with jitter, rotated exit IP	CAPTCHA interstitials, rate-limit blocks	empty shells and soft blocks — the block is deterministic, retry reproduces it
Quarantine for review	park the URL and the response, flag for human/mannual triage	empty shells, soft blocks, unknown failure modes	high-volume transient challenges — your quarantine queue will outgrow your review capacity

Quarantine deserves a design note: store the full response body alongside the URL, not just the verdict. When you’re staring at 400 quarantined store.example.com URLs, being able to open one and see the actual page is the difference between a ten-minute diagnosis and an afternoon of guessing. Disk is cheap; re-fetching evidence after the fact is not, because the block page you’d re-fetch may no longer be served.

One more thing. The retry path above says rotate_exit_ip=True for a reason: retrying a CAPTCHA through the same exit IP is asking the same bouncer to check the same ID. Whether that rotation is your own proxy pool or your provider’s managed rotation, the principle holds — the retry must arrive looking different, or don’t bother sending it.

Wrapping Up

A 200 means the server talked to you. It says nothing about what it said. The fix is a pipeline that treats every response as unproven until it passes checks at three levels: cheap fingerprints (title, body size, hash of the first bytes), structural contracts (the selectors and element counts that define a real page), and an escalation tier for JS challenges that only renders what’s flagged. Then log the verdict as first-class data and wire alerting to the detection rate, so contamination shows up on a dashboard instead of in a stakeholder’s spreadsheet three weeks later.

The pieces are small. A title regex, a size threshold, a selector count, a retry policy. What makes it work is that they all share one definition of success — the content contract — and that every fetch, at every tier, gets judged against it. Start with the fingerprint check and the log record; those two alone will tell you within a day how many of your “successful” fetches have been lying to you.

If you want to go deeper on the adjacent pieces: measure your scraper’s real success rate locally before trusting any of these numbers, and our client-side versus service-side retry comparison covers where the backoff logic should live.