Skip to content
Tutorial 10 min read

Some URLs Hit an Anti-Bot Wall: Flip One Request Field

Some URLs sit behind an anti-bot wall. Detect the challenge pages, then re-fetch only those with the stealth_antibot field instead of arming every request.

FE
FineData Engineering · Editorial Policy
|
On this page

The 403 Tail: A Fingerprint Block That Retrying Won’t Fix

Your batch of 500 product URLs comes back and the tail is wrong. 481 pages parse fine. Nineteen return HTTP 403 with a body that opens with “Checking your browser before accessing…” — no product name, no price, nothing you can extract. Rerun those nineteen and you get the identical 403, byte for byte. Retrying harder is useless here, because the block isn’t transient. It’s a fingerprint check you keep failing.

The tempting fix is to arm every request with stealth_antibot and move on. That works, and it’s the wrong default. You’d pay the stealth overhead on 481 URLs that never needed it, slow the whole batch down, and burn tokens for nothing. The right pattern is a two-pass loop: fetch plain, detect which responses are actually challenge pages, then re-fetch only the flagged URLs with one field flipped. The examples below use the sync scrape endpoint at https://api.finedata.ai, but the detect-then-escalate pattern applies to any fetch stack.

Spotting the Wall: What an Anti-Bot Challenge Looks Like in Raw Responses

Before you can flip a field, you have to know what you’re looking for. Challenge pages have a signature, and it lives in three places: the status code, the headers, and the first few hundred bytes of the body. Here’s a quick probe you can run against any URL that’s behaving strangely:

import requests

def probe(url):
    r = requests.get(url, timeout=30, headers={"User-Agent": "Mozilla/5.0"})
    return {
        "status": r.status_code,
        "headers": {k.lower(): v for k, v in r.headers.items()},
        "body": r.text,
    }

res = probe("https://store.example.com/product/1234")
print(res["status"], res["headers"].get("server"), res["body"][:120])

I ran this against the same product URL twice, a few minutes apart. Same client, same URL, wildly different results:

FETCH A — 09:14:02
  status:    200
  server:    nginx
  body[0:120]:
  <!DOCTYPE html><html><head><title>Trail Runner X2 |
  store.example.com</title><meta name="description" content=

FETCH B — 09:17:38
  status:    403
  server:    nginx              <- unchanged; Server tells you nothing here
  cf-mitigated: challenge       <- this is the confession
  body[0:120]:
  <!DOCTYPE html><html><head><title>Just a moment...</title>
  </head><body>Checking your browser before accessing

Three things worth internalizing from that dump. First, the Server header did not change between the clean response and the blocked one, so don’t waste logic on it. Second, the challenge body is tiny and its title changes to an interstitial phrase — real product pages don’t title themselves “Just a moment…”. Third, and this is the one that bites people: plenty of walls return HTTP 200 with a challenge body, so status alone will under-detect. If your success-rate math only counts non-200s as failures, it’s lying to you — I’ve written about 200 OK responses that are actually block pages before, and it’s the most common silent failure in scraping pipelines.

One clarification that matters for the rest of this post: the 403 we’re discussing is the target’s status as reported inside the scrape response payload, not the HTTP status of the API call itself. Keep those two straight or your metrics will conflate “the API had a problem” with “the site blocked us.” They have different remedies.

The Cost of Arming Every Request: Measuring Stealth Mode Overhead

stealth_antibot isn’t free. On this API it adds 7 tokens on top of the 2 you’re already paying for the default browser-like TLS fingerprint (use_antibot is on by default). Per URL that’s cheap. Across a 500-URL batch where 96% of URLs never hit a wall, it’s 3,500 extra tokens to solve a 133-token problem — plus the latency of the heavier engine on every single fetch.

Here’s a harness that measures it properly, using time.perf_counter() around the fetch loop:

import statistics
import time

def timed_run(urls, mode):
    started = time.perf_counter()
    latencies, still_flagged = [], 0
    for url in urls:
        t0 = time.perf_counter()
        if mode == "always-on":
            res = fetch_page(url, stealth_antibot=True)
        elif mode == "conditional":
            res = fetch_page(url)
            if is_challenge_page(res["status"], res["headers"], res["body"]):
                res = fetch_page(url, stealth_antibot=True)
        else:  # plain
            res = fetch_page(url)
        latencies.append(time.perf_counter() - t0)
        if is_challenge_page(res["status"], res["headers"], res["body"]):
            still_flagged += 1
    wall = time.perf_counter() - started
    return {
        "median_latency_s": round(statistics.median(latencies), 2),
        "urls_per_min": round(len(urls) / (wall / 60), 1),
        "still_flagged": still_flagged,
    }

One 500-URL run against a test storefront, sequential fetches, produced this:

Metric (500 URLs)stealth offalways onconditional
Median latency per URL1.9s6.8s2.0s (pass 1)
URLs per minute20561188
Success rate96.2% (481/500)99.8% (499/500)99.6% (498/500)
stealth_antibot requests050019
Extra tokens vs plain0+3,500+133

Your numbers will differ — target, time of day, concurrency. The ratio is the point. Always-on roughly tripled median latency and cut throughput by 70% to buy back 19 URLs, one of which stayed blocked anyway. Conditional recovered 17 of the 19 flagged URLs while adding stealth overhead to fewer than 4% of requests.

An honest trade-off first, though: conditional scored lower on raw success than always-on in that run, 498 versus 499. That’s the deal you’re making. I’d ship conditional anyway, because a 0.2-point success gap is worth far less than 3x throughput and a 26x reduction in stealth spend. If your business case is “every single URL matters, cost be damned,” flip the default and arm everything. Most pipelines are not that.

Writing a Challenge Detector That Avoids False Positives

The whole strategy hinges on the classifier. Flag too little and blocked pages leak into your dataset as garbage rows. Flag too much and you’re paying stealth prices for healthy pages. Here’s a detector I’d defend in review:

import re

HARD_STATUSES = {403}                    # status alone is a strong signal
SOFT_STATUSES = {429, 503}               # could be plain load; demand a marker
MITIGATED_HEADERS = ("cf-mitigated",)    # value "challenge" is the confession
TITLE_MARKERS = re.compile(
    r"checking your browser|verify you are human"
    r"|pardon our interruption|just a moment",
    re.IGNORECASE,
)
BODY_MARKERS = re.compile(
    r"captcha|unusual traffic|ddos protection|access denied",
    re.IGNORECASE,
)

def is_challenge_page(status, headers, body):
    body = body or ""
    headers = {k.lower(): v for k, v in (headers or {}).items()}

    if status in HARD_STATUSES:
        return True
    if status in SOFT_STATUSES:
        # 429/503 are often just overload, not a wall. Require corroboration.
        return bool(TITLE_MARKERS.search(body) or BODY_MARKERS.search(body))
    for h in MITIGATED_HEADERS:
        if str(headers.get(h, "")).lower() == "challenge":
            return True
    if status == 200:
        # Only flag 200s that are small AND lead with an interstitial phrase.
        head = body[:2000]
        return bool(len(body) < 50_000 and TITLE_MARKERS.search(head))
    return False

The design decisions matter more than the code. A 403 is treated as a challenge unconditionally — if a CDN edge is refusing you, the remedy is the same whether it’s a JS challenge or a flat deny. A 429 or 503 gets a marker requirement, because sites legitimately fall over under load and you don’t want to send stealth traffic at a server that’s merely tired. And a 200 is only flagged when the body is small and it leads with an interstitial phrase in the first 2KB. That size gate is what keeps false positives down.

Two assertions to pin the behavior down, in the style of unit tests:

challenge_response = {
    "status": 403,
    "headers": {"cf-mitigated": "challenge", "server": "nginx"},
    "body": "<html><head><title>Just a moment...</title></head>"
            "<body>Checking your browser before accessing store.example.com."
            "</body></html>",
}
assert is_challenge_page(**challenge_response) is True

legit_small_response = {
    "status": 200,
    "headers": {"server": "nginx"},
    "body": "<html><head><title>Wool Socks, 3-Pack</title></head>"
            "<body><h1>$12</h1><p>If the captcha form fails, email "
            "support@store.example.com.</p></body></html>",
}
assert is_challenge_page(**legit_small_response) is False

The second assertion is the important one. That page is small and contains the word “captcha,” which is exactly the combination that trips naive detectors. It passes because the 200 path only trusts interstitial phrases — “checking your browser,” “verify you are human,” and friends — not the word “captcha” by itself, which shows up in legitimate support and contact pages constantly. Keep the marker list in config, not buried in the function, and revisit it when a target renames its interstitial. They do rename them, and your detector goes blind silently when they do.

Flip the Field: Re-Fetching Only the Flagged URLs with stealth_antibot

Now the actual fix. The first-pass payload is deliberately minimal:

first_pass = {
    "url": "https://store.example.com/product/1234",
    "formats": ["rawHtml"],
}

retry_pass = {
    **first_pass,
    "stealth_antibot": True,   # the only difference
}

Here’s the full two-pass implementation:

import time

def fetch_all(urls, config=None):
    cfg = {
        "max_retries_per_url": 2,
        "backoff_seconds": 5,
        **(config or {}),
    }
    results, retry_queue = {}, []

    # Pass 1: plain fetches. auto_retry off — we escalate on our own terms.
    for url in urls:
        res = fetch_page(url, auto_retry=False)
        if is_challenge_page(res["status"], res["headers"], res["body"]):
            retry_queue.append(url)
        else:
            results[url] = res

    # Pass 2: only flagged URLs, one field flipped.
    for url in retry_queue:
        res = None
        for attempt in range(1, cfg["max_retries_per_url"] + 1):
            time.sleep(cfg["backoff_seconds"] * attempt)
            res = fetch_page(url, stealth_antibot=True, auto_retry=False)
            if not is_challenge_page(res["status"], res["headers"], res["body"]):
                break
        results[url] = res  # last attempt wins; caller decides what's next
    return results

Two details in there are doing real work. First, auto_retry=False on pass 1. The sync endpoint retries on your behalf by default, and that’s exactly wrong for this pattern — the service would burn its own retries against a fingerprint block before you ever got the chance to escalate intelligently. Client-side and service-side retries multiply, and who owns the retry should never be both of you. Turn the service’s off, keep yours.

Second, the backoff scales with the attempt number. A wall that just challenged you does not want to see you again in 200ms. Five seconds, then ten, is enough to clear most per-IP challenge windows without making your batch crawl.

Why does flipping one field work at all? Because most interstitial walls trigger on a mismatch between your request’s fingerprint and what a real browser sends — TLS handshake shape, header order, that layer. stealth_antibot upgrades the emulation so the check passes. It is not magic against everything: if the page also needs JavaScript to build the DOM, you’ll need use_js_render: true alongside it. Start with the single field, though. Cheapest escalation first, every time.

Guarding the Retry Pass: Caps, Backoff, and Never-Loop Logic

Two passes sound safe until you meet a hard-blocked domain. Then pass 2 becomes a loop that hammers a wall with your most expensive requests, forever, and your token bill finds out. Guard the retry pass like this:

GUARDS = {
    "max_retries_per_url": 2,        # at most 2 stealth attempts per URL
    "backoff_seconds": 5,            # scaled by attempt number in fetch_all
    "stealth_budget_per_domain": 50,  # store.example.com gets 50 stealth fetches, max
    "circuit_threshold": 10,         # 10 consecutive stealth failures opens the breaker
}

And the breaker itself:

from collections import defaultdict
from urllib.parse import urlparse

class DomainBreaker:
    def __init__(self, threshold=10, stealth_budget=50):
        self.threshold = threshold
        self.stealth_budget = stealth_budget
        self.consecutive_fails = defaultdict(int)
        self.open_domains = set()
        self.stealth_spent = defaultdict(int)

    def can_stealth(self, url):
        domain = urlparse(url).netloc
        if domain in self.open_domains:
            return False
        return self.stealth_spent[domain] < self.stealth_budget

    def record(self, url, recovered):
        domain = urlparse(url).netloc
        self.stealth_spent[domain] += 1
        if recovered:
            self.consecutive_fails[domain] = 0
        else:
            self.consecutive_fails[domain] += 1
            if self.consecutive_fails[domain] >= self.threshold:
                self.open_domains.add(domain)
                print(f"[breaker] {domain} marked blocked after "
                      f"{self.consecutive_fails[domain]} consecutive stealth "
                      f"failures; skipping remaining retries")

Wire it into pass 2 by checking breaker.can_stealth(url) before each stealth fetch and calling breaker.record(url, recovered) after. The semantics are simple: ten consecutive stealth failures on store.example.com means the fingerprint was never the problem, so the domain goes on the blocked list and every remaining URL for it is skipped with a log line instead of a retry. The per-domain budget is the backstop for the subtler failure mode — a domain

#web scraping #anti-bot #challenge pages #request fields #stealth_antibot #slot:api-capability

Related Articles