Some URLs Hit an Anti-Bot Wall: Flip One Request Field
Some URLs sit behind an anti-bot wall. Detect the challenge pages, then re-fetch only those with the stealth_antibot field instead of arming every request.
On this page
The 403 Tail: A Fingerprint Block That Retrying Won’t Fix
Your batch of 500 product URLs comes back and the tail is wrong. 481 pages parse fine. Nineteen return HTTP 403 with a body that opens with “Checking your browser before accessing…” — no product name, no price, nothing you can extract. Rerun those nineteen and you get the identical 403, byte for byte. Retrying harder is useless here, because the block isn’t transient. It’s a fingerprint check you keep failing.
The tempting fix is to arm every request with stealth_antibot and move on. That works, and it’s the wrong default. You’d pay the stealth overhead on 481 URLs that never needed it, slow the whole batch down, and burn tokens for nothing. The right pattern is a two-pass loop: fetch plain, detect which responses are actually challenge pages, then re-fetch only the flagged URLs with one field flipped. The examples below use the sync scrape endpoint at https://api.finedata.ai, but the detect-then-escalate pattern applies to any fetch stack.
Spotting the Wall: What an Anti-Bot Challenge Looks Like in Raw Responses
Before you can flip a field, you have to know what you’re looking for. Challenge pages have a signature, and it lives in three places: the status code, the headers, and the first few hundred bytes of the body. Here’s a quick probe you can run against any URL that’s behaving strangely:
import requests
def probe(url):
r = requests.get(url, timeout=30, headers={"User-Agent": "Mozilla/5.0"})
return {
"status": r.status_code,
"headers": {k.lower(): v for k, v in r.headers.items()},
"body": r.text,
}
res = probe("https://store.example.com/product/1234")
print(res["status"], res["headers"].get("server"), res["body"][:120])
I ran this against the same product URL twice, a few minutes apart. Same client, same URL, wildly different results:
FETCH A — 09:14:02
status: 200
server: nginx
body[0:120]:
<!DOCTYPE html><html><head><title>Trail Runner X2 |
store.example.com</title><meta name="description" content=
FETCH B — 09:17:38
status: 403
server: nginx <- unchanged; Server tells you nothing here
cf-mitigated: challenge <- this is the confession
body[0:120]:
<!DOCTYPE html><html><head><title>Just a moment...</title>
</head><body>Checking your browser before accessing
Three things worth internalizing from that dump. First, the Server header did not change between the clean response and the blocked one, so don’t waste logic on it. Second, the challenge body is tiny and its title changes to an interstitial phrase — real product pages don’t title themselves “Just a moment…”. Third, and this is the one that bites people: plenty of walls return HTTP 200 with a challenge body, so status alone will under-detect. If your success-rate math only counts non-200s as failures, it’s lying to you — I’ve written about 200 OK responses that are actually block pages before, and it’s the most common silent failure in scraping pipelines.
One clarification that matters for the rest of this post: the 403 we’re discussing is the target’s status as reported inside the scrape response payload, not the HTTP status of the API call itself. Keep those two straight or your metrics will conflate “the API had a problem” with “the site blocked us.” They have different remedies.
The Cost of Arming Every Request: Measuring Stealth Mode Overhead
stealth_antibot isn’t free. On this API it adds 7 tokens on top of the 2 you’re already paying for the default browser-like TLS fingerprint (use_antibot is on by default). Per URL that’s cheap. Across a 500-URL batch where 96% of URLs never hit a wall, it’s 3,500 extra tokens to solve a 133-token problem — plus the latency of the heavier engine on every single fetch.
Here’s a harness that measures it properly, using time.perf_counter() around the fetch loop:
import statistics
import time
def timed_run(urls, mode):
started = time.perf_counter()
latencies, still_flagged = [], 0
for url in urls:
t0 = time.perf_counter()
if mode == "always-on":
res = fetch_page(url, stealth_antibot=True)
elif mode == "conditional":
res = fetch_page(url)
if is_challenge_page(res["status"], res["headers"], res["body"]):
res = fetch_page(url, stealth_antibot=True)
else: # plain
res = fetch_page(url)
latencies.append(time.perf_counter() - t0)
if is_challenge_page(res["status"], res["headers"], res["body"]):
still_flagged += 1
wall = time.perf_counter() - started
return {
"median_latency_s": round(statistics.median(latencies), 2),
"urls_per_min": round(len(urls) / (wall / 60), 1),
"still_flagged": still_flagged,
}
One 500-URL run against a test storefront, sequential fetches, produced this:
| Metric (500 URLs) | stealth off | always on | conditional |
|---|---|---|---|
| Median latency per URL | 1.9s | 6.8s | 2.0s (pass 1) |
| URLs per minute | 205 | 61 | 188 |
| Success rate | 96.2% (481/500) | 99.8% (499/500) | 99.6% (498/500) |
| stealth_antibot requests | 0 | 500 | 19 |
| Extra tokens vs plain | 0 | +3,500 | +133 |
Your numbers will differ — target, time of day, concurrency. The ratio is the point. Always-on roughly tripled median latency and cut throughput by 70% to buy back 19 URLs, one of which stayed blocked anyway. Conditional recovered 17 of the 19 flagged URLs while adding stealth overhead to fewer than 4% of requests.
An honest trade-off first, though: conditional scored lower on raw success than always-on in that run, 498 versus 499. That’s the deal you’re making. I’d ship conditional anyway, because a 0.2-point success gap is worth far less than 3x throughput and a 26x reduction in stealth spend. If your business case is “every single URL matters, cost be damned,” flip the default and arm everything. Most pipelines are not that.
Writing a Challenge Detector That Avoids False Positives
The whole strategy hinges on the classifier. Flag too little and blocked pages leak into your dataset as garbage rows. Flag too much and you’re paying stealth prices for healthy pages. Here’s a detector I’d defend in review:
import re
HARD_STATUSES = {403} # status alone is a strong signal
SOFT_STATUSES = {429, 503} # could be plain load; demand a marker
MITIGATED_HEADERS = ("cf-mitigated",) # value "challenge" is the confession
TITLE_MARKERS = re.compile(
r"checking your browser|verify you are human"
r"|pardon our interruption|just a moment",
re.IGNORECASE,
)
BODY_MARKERS = re.compile(
r"captcha|unusual traffic|ddos protection|access denied",
re.IGNORECASE,
)
def is_challenge_page(status, headers, body):
body = body or ""
headers = {k.lower(): v for k, v in (headers or {}).items()}
if status in HARD_STATUSES:
return True
if status in SOFT_STATUSES:
# 429/503 are often just overload, not a wall. Require corroboration.
return bool(TITLE_MARKERS.search(body) or BODY_MARKERS.search(body))
for h in MITIGATED_HEADERS:
if str(headers.get(h, "")).lower() == "challenge":
return True
if status == 200:
# Only flag 200s that are small AND lead with an interstitial phrase.
head = body[:2000]
return bool(len(body) < 50_000 and TITLE_MARKERS.search(head))
return False
The design decisions matter more than the code. A 403 is treated as a challenge unconditionally — if a CDN edge is refusing you, the remedy is the same whether it’s a JS challenge or a flat deny. A 429 or 503 gets a marker requirement, because sites legitimately fall over under load and you don’t want to send stealth traffic at a server that’s merely tired. And a 200 is only flagged when the body is small and it leads with an interstitial phrase in the first 2KB. That size gate is what keeps false positives down.
Two assertions to pin the behavior down, in the style of unit tests:
challenge_response = {
"status": 403,
"headers": {"cf-mitigated": "challenge", "server": "nginx"},
"body": "<html><head><title>Just a moment...</title></head>"
"<body>Checking your browser before accessing store.example.com."
"</body></html>",
}
assert is_challenge_page(**challenge_response) is True
legit_small_response = {
"status": 200,
"headers": {"server": "nginx"},
"body": "<html><head><title>Wool Socks, 3-Pack</title></head>"
"<body><h1>$12</h1><p>If the captcha form fails, email "
"support@store.example.com.</p></body></html>",
}
assert is_challenge_page(**legit_small_response) is False
The second assertion is the important one. That page is small and contains the word “captcha,” which is exactly the combination that trips naive detectors. It passes because the 200 path only trusts interstitial phrases — “checking your browser,” “verify you are human,” and friends — not the word “captcha” by itself, which shows up in legitimate support and contact pages constantly. Keep the marker list in config, not buried in the function, and revisit it when a target renames its interstitial. They do rename them, and your detector goes blind silently when they do.
Flip the Field: Re-Fetching Only the Flagged URLs with stealth_antibot
Now the actual fix. The first-pass payload is deliberately minimal:
first_pass = {
"url": "https://store.example.com/product/1234",
"formats": ["rawHtml"],
}
retry_pass = {
**first_pass,
"stealth_antibot": True, # the only difference
}
Here’s the full two-pass implementation:
import time
def fetch_all(urls, config=None):
cfg = {
"max_retries_per_url": 2,
"backoff_seconds": 5,
**(config or {}),
}
results, retry_queue = {}, []
# Pass 1: plain fetches. auto_retry off — we escalate on our own terms.
for url in urls:
res = fetch_page(url, auto_retry=False)
if is_challenge_page(res["status"], res["headers"], res["body"]):
retry_queue.append(url)
else:
results[url] = res
# Pass 2: only flagged URLs, one field flipped.
for url in retry_queue:
res = None
for attempt in range(1, cfg["max_retries_per_url"] + 1):
time.sleep(cfg["backoff_seconds"] * attempt)
res = fetch_page(url, stealth_antibot=True, auto_retry=False)
if not is_challenge_page(res["status"], res["headers"], res["body"]):
break
results[url] = res # last attempt wins; caller decides what's next
return results
Two details in there are doing real work. First, auto_retry=False on pass 1. The sync endpoint retries on your behalf by default, and that’s exactly wrong for this pattern — the service would burn its own retries against a fingerprint block before you ever got the chance to escalate intelligently. Client-side and service-side retries multiply, and who owns the retry should never be both of you. Turn the service’s off, keep yours.
Second, the backoff scales with the attempt number. A wall that just challenged you does not want to see you again in 200ms. Five seconds, then ten, is enough to clear most per-IP challenge windows without making your batch crawl.
Why does flipping one field work at all? Because most interstitial walls trigger on a mismatch between your request’s fingerprint and what a real browser sends — TLS handshake shape, header order, that layer. stealth_antibot upgrades the emulation so the check passes. It is not magic against everything: if the page also needs JavaScript to build the DOM, you’ll need use_js_render: true alongside it. Start with the single field, though. Cheapest escalation first, every time.
Guarding the Retry Pass: Caps, Backoff, and Never-Loop Logic
Two passes sound safe until you meet a hard-blocked domain. Then pass 2 becomes a loop that hammers a wall with your most expensive requests, forever, and your token bill finds out. Guard the retry pass like this:
GUARDS = {
"max_retries_per_url": 2, # at most 2 stealth attempts per URL
"backoff_seconds": 5, # scaled by attempt number in fetch_all
"stealth_budget_per_domain": 50, # store.example.com gets 50 stealth fetches, max
"circuit_threshold": 10, # 10 consecutive stealth failures opens the breaker
}
And the breaker itself:
from collections import defaultdict
from urllib.parse import urlparse
class DomainBreaker:
def __init__(self, threshold=10, stealth_budget=50):
self.threshold = threshold
self.stealth_budget = stealth_budget
self.consecutive_fails = defaultdict(int)
self.open_domains = set()
self.stealth_spent = defaultdict(int)
def can_stealth(self, url):
domain = urlparse(url).netloc
if domain in self.open_domains:
return False
return self.stealth_spent[domain] < self.stealth_budget
def record(self, url, recovered):
domain = urlparse(url).netloc
self.stealth_spent[domain] += 1
if recovered:
self.consecutive_fails[domain] = 0
else:
self.consecutive_fails[domain] += 1
if self.consecutive_fails[domain] >= self.threshold:
self.open_domains.add(domain)
print(f"[breaker] {domain} marked blocked after "
f"{self.consecutive_fails[domain]} consecutive stealth "
f"failures; skipping remaining retries")
Wire it into pass 2 by checking breaker.can_stealth(url) before each stealth fetch and calling breaker.record(url, recovered) after. The semantics are simple: ten consecutive stealth failures on store.example.com means the fingerprint was never the problem, so the domain goes on the blocked list and every remaining URL for it is skipped with a log line instead of a retry. The per-domain budget is the backstop for the subtler failure mode — a domain
Related Articles
Scrape Mobile Pages Through a Carrier Exit IP
Some sites serve challenges to desktop traffic but real pages to phones. Learn when a mobile carrier exit IP, set with use_mobile, is the fix.
TutorialScrape Localized Storefronts With a Country Exit Code
Storefronts change price, language, and stock by visitor country. Pin the scrape exit with an ISO-2 code and handle 422 when the country is unsupported.
TutorialSticky Exit IPs for Warmup and Follow-Up Scrapes
Rotating proxies between a warmup hit and the next request breaks cookies and geo checks. Learn how a sticky exit IP keeps the session on the same IP.