Too Many Redirects: Fix Redirect Loops in Your Scraper

When the Server Isn’t Actually Broken

Your scraper hits a page at 3 AM and dies with requests.exceptions.TooManyRedirects: Exceeded 30 redirects. The same URL loads instantly in your browser. No error, no challenge page, just a normal page. You reload it in an incognito window — still fine. So the site is up, the URL is valid, and your scraper is the only client that can’t fetch it.

This is one of the most misleading failures in web scraping, because the error message blames the server (“too many redirects”) while the actual cause is almost always something your scraper is not sending. Redirect loops are a negotiation between the client and the server. Browsers negotiate well. Scrapers negotiate badly.

Here’s the shape of the problem, how to reproduce it, and how to fix both major causes — cookies and geography.

Why ‘TooManyRedirects’ Only Happens in Your Scraper, Not Your Browser

A redirect loop means the server keeps telling you “go somewhere else,” and wherever you land tells you to go back. The server isn’t broken for real users because real users carry state that scrapers drop. The browser has cookies from previous visits, a consistent set of headers, and an IP address that matches its Accept-Language. Your scraper has none of that.

Compare what actually arrives at store.example.com from each client:

Header	Browser	Bare requests.get()
Cookie	sid=abc123; locale=en-us; cart_v2=...	(absent)
User-Agent	Real Chrome UA	python-requests/2.x
Accept-Language	en-US,en;q=0.9	(absent)
Referer	https://store.example.com/	(absent)
Accept	text/html,application/xhtml+xml,...	*/*

The Cookie row is the killer. Many storefronts issue a session cookie on first contact and redirect you once so the cookie gets set on the right path or subdomain. If the client doesn’t persist that cookie, the server sees a “new” visitor every hop, issues a new cookie, and redirects again. Forever.

The loop typically looks like this:

GET  https://store.example.com/account
301  -> https://store.example.com/login?next=/account

GET  https://store.example.com/login?next=/account
     (server sets a fresh Set-Cookie, but scraper drops it)
302  -> https://store.example.com/account   # "you're already logged in... maybe"

GET  https://store.example.com/account
     (no session cookie -> not authorized)
301  -> https://store.example.com/login?next=/account

... repeat until the client's redirect limit trips

Three hops, then it cycles. Browsers never see this because after the first hop they carry the Set-Cookie value forward and the cycle breaks. requests.get() called in a loop creates a new connection context every time — each call is a stranger knocking on the door.

There’s a second flavor that has nothing to do with auth: geo-redirects. A site sees your datacenter IP in one country, your Accept-Language claiming another, and its own default store region in a third — and bounces you between /us/ and /de/ storefronts trying to reconcile them. We’ll cover that after the cookie case, because the diagnosis is different.

One quick sanity check before anything else: open DevTools, Network tab, “Preserve log” enabled, and load the URL in your browser. If your browser follows one or two redirects max, the site is fine and the problem is client-side. If the browser also loops, you’ve got a genuinely broken site or you’re already cookied into a bad state — clear cookies and retest.

Reproducing the Loop: Capturing the Redirect Chain with curl and Python

Don’t guess. Observe. Before touching your scraper, capture the exact hop sequence so you know which URLs participate in the loop and what status codes connect them.

With curl:

curl -v --max-redirs 5 -L https://store.example.com/account 2>&1 | grep -E '^(< HTTP|< location|\* Connected)'

Typical output for a loop:

< HTTP/1.1 301 Moved Permanently
< location: https://store.example.com/login?next=/account
< HTTP/1.1 302 Found
< location: https://store.example.com/account
< HTTP/1.1 301 Moved Permanently
< location: https://store.example.com/login?next=/account
curl: (47) Maximum (5) redirects followed.

That’s your confirmation. The chain is /account -> /login -> /account -> /login -> .... Note whether the codes are 301, 302, or 307 — a 307/308 preserves the method and body, which matters if you’re scraping via POST. A 301/302 on a POST will often get downgraded to GET by the client library, which can itself cause the loop if the server expected the method to survive.

Now the same thing in Python, hop by hop, so you can see every Set-Cookie along the way:

import requests

url = "https://store.example.com/account"
seen = set()
hop = 0

while url and url not in seen and hop < 10:
    seen.add(url)
    r = requests.get(url, allow_redirects=False, timeout=15)
    print(f"hop {hop}: {r.status_code} {url}")
    for set_cookie in r.raw.headers.getlist("Set-Cookie"):
        print(f"        Set-Cookie: {set_cookie}")
    url = r.headers.get("Location")
    hop += 1

if url in seen:
    print(f"LOOP DETECTED: {url} already visited")

Expected output for the cookie case:

hop 0: 301 https://store.example.com/account
        Set-Cookie: sid=8f3a...; Path=/; HttpOnly
hop 1: 302 https://store.example.com/login?next=/account
hop 2: 301 https://store.example.com/account
LOOP DETECTED: https://store.example.com/login?next=/account already visited

Look at hop 0. The server set a session cookie, and hop 1 immediately asked you back to /account — the “you’re logged in” redirect. But hop 2 got a 301 back to login, which means the server did not see the cookie on hop 2. It was set, then dropped. That’s your diagnosis, in five lines of output.

If instead you see a bounce between regional paths (/us/, /de/) with no meaningful Set-Cookie, skip ahead to the geo section.

Cookie-Based Loops: The Session Cookie Your Scraper Discards

Here’s the broken pattern, and I’ve seen it in production code more times than I can count:

import requests

urls = [
    "https://store.example.com/account",
    "https://store.example.com/orders",
    "https://store.example.com/settings",
]

for url in urls:
    r = requests.get(url, timeout=15)  # new context, cookie jar discarded every call
    print(url, r.status_code)

Every requests.get() at module level uses a throwaway context. The Set-Cookie from the first response evaporates. Worse, this code often works for weeks until the site adds a session check — then it loops, and nobody connects the change to the failure.

The fix is a Session:

import requests

s = requests.Session()
s.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Accept-Language": "en-US,en;q=0.9",
})

# First request: server sets the session cookie
r1 = s.get("https://store.example.com/account", timeout=15)
print(r1.status_code, dict(s.cookies))  # {'sid': '8f3a...', 'locale': 'en-us'}

# Second request: cookie jar is sent automatically
r2 = s.get("https://store.example.com/orders", timeout=15)
print(r2.status_code, len(r2.text))

The first response’s Set-Cookie: sid=8f3a...; Path=/; HttpOnly now lands in s.cookies, and the second request carries Cookie: sid=8f3a... — exactly like the browser did in the table above. The loop breaks because the server finally recognizes the visitor across hops.

Two subtleties worth knowing:

Cookie expiry mid-crawl. Sessions die. If your crawl runs for hours and the sid expires, you can loop again even with a Session. Watch for a sudden redirect to /login on a URL that worked earlier, and re-establish the session when you see it.

Domain-scoped cookies. If the loop spans subdomains (example.com -> store.example.com), a cookie set with Domain=.example.com is fine, but one set on store.example.com alone won’t travel back to the parent. Inspect the Domain attribute in the Set-Cookie header from your hop-by-hop trace. This is rare, but when it bites, it’s maddening.

Geo- and Language-Based Redirects: When the Server Bounces You by Region

Cookie loops are an auth problem. Geo loops are an identity problem. The server looks at your IP, your Accept-Language, and its own default region, gets three different answers, and ping-pongs you:

GET https://example.com/us/products
    (server sees datacenter IP geolocated to DE,
     Accept-Language absent -> assumes German visitor)
302 -> https://example.com/de/products

GET https://example.com/de/products
    (server's config or CDN edge says /us/ is the canonical
     storefront for this hostname)
302 -> https://example.com/us/products

... cycle

Neither hop involves auth. No Set-Cookie matters here. The fix is to make your identity consistent: pick one region, pin it in the URL, set the matching language header, and — if the site geolocates by IP — make sure your exit IP matches the region you’re claiming.

import requests

s = requests.Session()
s.max_redirects = 5
s.headers.update({
    "Accept-Language": "en-US,en;q=0.9",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
})

# Pin the regional path, don't let the server choose
r = s.get("https://store.example.com/en-us/products", timeout=15)

If the site still bounces you because your datacenter IP geolocates to the wrong country, you need a matching exit IP. This is where a scraping API earns its keep — you can request a specific country for the proxy exit, keep headers consistent, and let the service manage cookie persistence across its fetch:

import time
import requests

API = "https://api.finedata.ai/api/v1"
auth = {"Authorization": "Bearer fd_your_api_key"}

# Submit an async scrape: US exit IP, pinned locale, redirects not chased
resp = requests.post(
    f"{API}/async/scrape",
    headers=auth,
    json={
        "url": "https://example.com",
        "proxy_country": "US",          # exit IP that matches your claimed locale
        "headers": {"Accept-Language": "en-US,en;q=0.9"},
        "formats": ["markdown"],
        "allow_redirects": False,       # surface the 30x instead of following it
        "max_retries": 3,
    },
    timeout=180,
)
resp.raise_for_status()
job_id = resp.json()["job_id"]

# Poll job status until it reaches a terminal state
while True:
    job = requests.get(f"{API}/async/jobs/{job_id}", headers=auth, timeout=30).json()
    if job.get("status") in ("completed", "failed", "cancelled"):
        break
    time.sleep(2)

print(job.get("status"), job.get("data", {}).get("markdown", "")[:200])

For crawls where you must keep one IP across a multi-step sequence (region selection, then category browse, then product pages), session_id with session_ttl pins the same exit IP for the whole sequence — same idea as a sticky session cookie, but at the network layer. If you’re deciding whether running your own proxies is worth it, own proxy pool vs managed rotation covers the trade-offs.

My opinionated take: pin the region explicitly whenever you can. Scrapers that hit the bare example.com root and let the server redirect them to a storefront are fragile by construction — you’re outsourcing a routing decision to a server that may not even know your intent. Hardcode /en-us/ (or whatever region you need), set the header, and the redirect never happens. Fewer redirects means fewer ways to fail. We cover the geo angle in more depth in scraping localized storefronts with a country exit code.

Capping Redirects: The Safety Net Every Scraper Needs

Even after you fix the root cause, keep a hard redirect limit. Loops have a way of resurfacing after a site redesign, and an uncapped scraper doesn’t just fail — it burns bandwidth, hammers the target, and can take hours to die if the redirect limit is high and your timeout is generous.

The defaults differ more than you’d expect:

Library	Setting	Default	Recommended
requests	session.max_redirects	30	5
httpx	follow_redirects + manual loop, or max_redirects	20	5
Scrapy	REDIRECT_MAX_TIMES (settings)	20	3–5

Thirty redirects is an absurd default for scraping. A legitimate page that needs more than five hops is a site with a redirect problem of its own, and you probably don’t want its HTML anyway.

import requests
from requests.exceptions import TooManyRedirects

s = requests.Session()
s.max_redirects = 5

try:
    r = s.get("https://store.example.com/account", timeout=15)
except TooManyRedirects:
    print(f"redirect cap hit on store.example.com/account, "
          f"cookie jar: {dict(s.cookies)}")
    raise
else:
    hops = len(r.history)
    print(f"ok after {hops} redirect(s), final URL: {r.url}")

Fail fast, log the jar state, and let your retry layer decide what to do. One caveat: don’t blindly retry a TooManyRedirects. Unlike a timeout or a 503, this error is deterministic — the same request will loop the same way. Retrying a redirect loop just multiplies the load. Circuit-break instead, and trip a breaker when the target degrades if loops start appearing across many URLs at once, because that pattern usually means a site change rather than a per-URL quirk.

Fixing the Loop: A Diagnostic Decision Flow for Cookie vs. Geo Causes

When a loop appears, run this sequence. It takes ten minutes and it always terminates in an answer:

1. Reproduce with the hop-by-hop trace from earlier. You need the exact URLs and status codes. No exceptions — you’re guessing otherwise.
2. Inspect every Set-Cookie in the chain. If a cookie is set on hop N and the server still treats hop N+1 as anonymous, it’s a cookie persistence problem. Fix: use a Session, or if it’s expiring mid-crawl, re-establish periodically.
3. Compare against DevTools. Open the Network tab with “Preserve log”, load the page in your browser, and diff the request headers on the first hop against what your scraper sent. Missing Cookie or Accept-Language? There’s your gap.
4. Check for regional URLs in the loop. /us/ <-> /de/ bouncing with no meaningful cookies means a geo mismatch. Fix: pin the regional URL, set Accept-Language, and verify your exit IP’s country.
5. Pin the session. One Session object per crawl target, region pinned, redirect cap on. If the target is hostile to datacenter IPs, route through an exit that matches your claimed locale.

And the assembled version — a small, complete fetch function that combines everything:

import time
import requests
from requests.exceptions import TooManyRedirects, RequestException

def fetch(url, session=None, max_attempts=3):
    s = session or requests.Session()
    s.max_redirects = 5
    if "Accept-Language" not in s.headers:
        s.headers.update({
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
            "Accept-Language": "en-US,en;q=0.9",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        })

    jar_reset = False
    for attempt in range(1, max_attempts + 1):
        try:
            r = s.get(url, timeout=15)
            if r.status_code == 200:
                return r
            if r.status_code in (429, 502, 503):
                time.sleep(2 ** attempt)   # exponential backoff
                continue
            r.raise_for_status()
        except TooManyRedirects:
            if jar_reset:
                # looped even with a clean jar -- deterministic, needs a human
                raise
            jar_reset = True               # allow exactly one fresh-jar retry
            s.cookies.clear()              # maybe a poisoned cookie caused the loop
            time.sleep(1)
            continue
        except RequestException:
            if attempt == max_attempts:
                raise
            time.sleep(2 ** attempt)
    raise RuntimeError(f"failed after {max_attempts} attempts: {url}")

r = fetch("https://store.example.com/en-us/products")
print(len(r.text), r.url)

Note the TooManyRedirects handler: it clears the cookie jar and retries once with a fresh state, on the theory that the jar itself may hold a poisoned cookie — the jar_reset flag limits the retry to a single attempt, and a second TooManyRedirects re-raises immediately. That’s the one redirect-loop case where a retry can help. If it loops twice in a row, it’s deterministic and needs human diagnosis.

Preventing Regressions: Logging Redirect Chains in Production

Fixing today’s loop is table stakes. The next one will arrive with a site update, and you want your logs to tell you which type it is before anyone opens a terminal.

requests gives you the full chain for free on every response via r.history:

import logging

logger = logging.getLogger("scraper.redirects")
ALERT_HOPS = 3

def log_chain(url, response):
    chain = [h.status_code for h in response.history]
    if len(chain) >= ALERT_HOPS:
        logger.warning(
            "redirect_chain url=%s hops=%d chain=%s final=%s",
            url, len(chain),
            " -> ".join(f"{h.status_code}:{h.url}" for h in response.history),
            response.url,
        )
    else:
        logger.info("redirect_chain url=%s hops=%d", url, len(chain), )

Healthy vs. degraded output:

# Healthy: language-selection redirect, one hop, expected
INFO  redirect_chain url=https://store.example.com/products hops=1

# Degraded: cap reached, loop confirmed, this URL needs attention
WARNING redirect_chain url=https://store.example.com/account hops=5 \
  chain=301:.../account -> 302:.../login?next=/account -> \
  301:.../account -> 302:.../login?next=/account -> 301:.../account \
  final=https://store.example.com/login?next=/account

The alert threshold of three is deliberate. One or two hops is normal site behavior — trailing-slash normalization, HTTP-to-HTTPS, language selection. Three or more on a URL that used to resolve directly is the earliest signal you’ll get that something changed. If you aggregate these warnings per domain, a spike across many URLs at once points to a site-side change; a single URL misbehaving points to a per-page quirk. That distinction saves you an hour of debugging by itself.

If you’re feeding scraped content to LLMs or downstream pipelines, remember that redirect handling also affects which page you actually parsed — a silent two-hop redirect to a login page returns HTTP 200 with useless HTML, which is its own failure mode and one we cover separately in your 200 OK responses are actually block pages.

Wrap-Up

Redirect loops are almost never the server’s fault, even though the error message reads that way. The browser succeeds because it carries state — cookies, consistent headers, a coherent identity. Your scraper loops because it drops that state between hops.

The playbook: reproduce with allow_redirects=False and a manual hop loop so you can see every Location and Set-Cookie. Diagnose from the trace — cookie set but ignored means a Session fix; regional URLs bouncing means a geo fix with pinned locale and matching exit IP. Cap redirects at five so loops fail in milliseconds instead of minutes. Log response.history on every request so the next regression announces itself.

Three-hop loops that cost you an afternoon usually come down to one missing Cookie header. Ten minutes of tracing beats hours of guessing.