Count the Hops: Measure Redirect Chains With a Local Tracer

The Invisible Cost of Redirect Chains

A scraper I was profiling recently looked broken: the logs said a single page took 2.4 seconds, but the page itself was static and fast. The culprit wasn’t the target page at all. It was the three redirects in front of it — a legacy URL, a host migration, and a trailing-slash normalization — each adding a full DNS lookup, TCP handshake, and TLS negotiation. Every HTTP library I’d used was quietly swallowing all of that into one opaque response.

If you let your client auto-follow redirects, you cannot see any of this. You get a final status code and a total wall-clock time, and the middle of the journey is a black box. This post builds a small tracer that disables auto-follow, walks each 3xx hop manually, and logs status, target, and timing per hop. Once you can count the hops, redirect chains stop being invisible latency and become something you can cache away.

Why Auto-Follow Redirects Hide the Real Cost of a Scrape

requests.get() follows redirects by default. So does urllib, httpx, curl, and every browser. For interactive use that’s the right default — users want the page, not the plumbing. For a scraper, it’s a measurement problem.

import requests, time

t0 = time.perf_counter()
resp = requests.get("https://example.com/product")
elapsed_ms = (time.perf_counter() - t0) * 1000

print(resp.status_code)     # 200
print(resp.url)             # https://store.example.com/products/123
print(f"{elapsed_ms:.0f} ms")  # 2,412 ms — for what exactly?

That request traversed three redirects. The response object knows the final URL via resp.url and even exposes resp.history — a list of the intermediate responses. But history gives you no timing per hop, no connection-reuse information, and it’s easy to forget it exists. Most scrapers I’ve audited never look at it.

Here’s what you actually know versus what actually happened:

	allow_redirects=True (auto-follow)	Manual hop-by-hop follow
Status codes seen	Only final (200)	Every hop: 301, 302, 308
Total time attributed	One opaque number	Per-hop elapsed ms
Intermediate targets	Buried in resp.history	First-class log entries
Cross-domain cookie drops	Invisible	Detectable per hop
Loop detection	Client’s internal limit (usually 30), then exception	Your own policy

The worst part is cost attribution. If you’re paying per scrape through a scraping API or a proxy provider, a three-hop chain can mean three proxied round trips billed as one page — or worse, one page billed at three times the expected latency budget. I covered this class of hidden cost in How Retries Quietly Double Your Scraping Costs, and redirect chains are the same disease with different symptoms.

Disabling Auto-Follow in Python HTTP Clients

Two libraries cover most Python scraping code. Both make this a one-parameter change, but the parameter names differ — a classic source of copy-paste bugs.

With requests:

import requests

# requests: the parameter is allow_redirects (verb form)
resp = requests.get(
    "https://example.com/login",
    allow_redirects=False,   # requests: allow_redirects
    timeout=10,
)
print(resp.status_code)      # 302
print(resp.headers.get("Location"))  # /session/new

With httpx:

import httpx

# httpx: the parameter is follow_redirects (no "allow", no verb form)
resp = httpx.get(
    "https://store.example.com/session",
    follow_redirects=False,  # httpx: follow_redirects — NOT allow_redirects
    timeout=10,
)
print(resp.status_code)
print(resp.headers.get("Location"))

One gotcha: httpx also has a module-level default you can set on the client (httpx.Client(follow_redirects=False)), which is the better choice if you’re building the tracer as a class. Passing the flag per-request works too, but defaults at the client level mean you can’t accidentally re-enable following in one code path.

Also note that requests treats a HEAD request with allow_redirects=False the same way — you can probe redirect chains cheaply with HEAD before committing to a full GET that downloads the body. Not all servers honor HEAD correctly, but when they do, it’s a fast way to map a chain without paying for content.

Parsing the Location Header Without Chasing It

Once auto-follow is off, every 3xx response hands you a Location header. Naive code assumes it’s an absolute URL. It frequently isn’t.

The Location header can legally be a relative path, a scheme-relative URL, or — on badly configured servers — missing entirely. Resolve it against the URL that produced the response:

from urllib.parse import urljoin

base = "https://store.example.com/products"
location = "/products/123"

resolved = urljoin(base, location)
print(resolved)  # https://store.example.com/products/123

urljoin handles the messy cases correctly, including the scheme-relative form where a redirect to //cdn.example.com/x silently switches you from HTTPS to whatever scheme the target host speaks. That switch is a security consideration (cookie leakage over plaintext) and a scraping consideration (the next hop may behave completely differently).

Location value	Base URL	Resolved result
https://cdn.example.com/assets	https://example.com/page	https://cdn.example.com/assets (absolute, used as-is)
/products/123	https://store.example.com/deals	https://store.example.com/products/123
//cdn.example.com/img	https://example.com/page	https://cdn.example.com/img (scheme inherited — check it)
(missing header)	any	No valid target — treat as a terminal failure, do not guess

The missing-header case deserves a hard rule: if a 3xx arrives with no Location, log it and stop. Some servers emit a Refresh meta tag or a JavaScript redirect instead, and guessing at those inside an HTTP-level tracer turns it into a browser. Keep the tracer HTTP-only; if the redirect lives in the page body, that’s a job for a JS-rendering scrape with a longer timeout, not more string parsing.

Building the Hop Logger: Status, Target, and Timing Per Redirect

Now the core piece. The tracer loops while the response is a 3xx, records one entry per hop, and returns the hop log plus the final response.

import time
import requests
from urllib.parse import urljoin

def trace_redirects(url, timeout=10, headers=None):
    """Follow redirects manually, logging status, target, and timing per hop."""
    session = requests.Session()
    hops = []
    current = url
    method = "GET"

    while True:
        t0 = time.perf_counter()
        resp = session.request(method, current, allow_redirects=False,
                               timeout=timeout, headers=headers)
        elapsed_ms = (time.perf_counter() - t0) * 1000

        if resp.status_code not in (301, 302, 303, 307, 308):
            return hops, resp

        location = resp.headers.get("Location")
        if not location:
            hops.append({"from": current, "status": resp.status_code,
                         "target": None, "elapsed_ms": round(elapsed_ms, 1),
                         "error": "3xx without Location header"})
            return hops, resp

        target = urljoin(current, location)
        hops.append({
            "from": current,
            "status": resp.status_code,
            "target": target,
            "elapsed_ms": round(elapsed_ms, 1),
        })

        # 303 always rewrites to GET; 301/302 rewrite POST to GET in practice
        if resp.status_code == 303 or (method == "POST"
                                       and resp.status_code in (301, 302)):
            method = "GET"

        current = target

Each entry is a dict, which makes the output trivial to serialize. Run it and dump the log as JSON lines:

{"from": "https://example.com/old-page", "status": 301, "target": "https://example.com/products/old-page", "elapsed_ms": 412.3}
{"from": "https://example.com/products/old-page", "status": 302, "target": "https://store.example.com/old-page", "elapsed_ms": 689.1}
{"from": "https://store.example.com/old-page", "status": 308, "target": "https://store.example.com/products/123", "elapsed_ms": 203.7}

Three hops, 1,305 ms of the total page time spent before any content arrived. The first hop is the expensive one — it’s a cold connection to a new host. The 308 at the end is a trailing-slash or path normalization, which is pure waste on every single scrape of that URL.

Guarding Against Loops and Runaway Chains

Auto-follow clients cap chains internally (requests stops at 30 hops and raises). Your tracer needs its own cap, plus loop detection — because A→B→A will happily spin forever under a naive while 3xx loop, and each iteration costs a real round trip.

class RedirectLoopError(Exception):
    def __init__(self, hops, message):
        self.hops = hops
        super().__init__(message)

def trace_redirects_guarded(url, max_hops=10, timeout=10):
    session = requests.Session()
    hops = []
    seen_urls = {url}
    current = url

    for _ in range(max_hops):
        t0 = time.perf_counter()
        resp = session.get(current, allow_redirects=False, timeout=timeout)
        elapsed_ms = (time.perf_counter() - t0) * 1000

        if resp.status_code not in (301, 302, 303, 307, 308):
            return hops, resp

        location = resp.headers.get("Location")
        if not location:
            return hops, resp

        target = urljoin(current, location)
        hops.append({"from": current, "status": resp.status_code,
                     "target": target, "elapsed_ms": round(elapsed_ms, 1)})

        if target in seen_urls:
            raise RedirectLoopError(
                hops,
                f"Redirect loop detected: {target} already visited",
            )
        seen_urls.add(target)
        current = target

    raise RedirectLoopError(hops, f"Exceeded max_hops={max_hops}")

Against a contrived cycle — example.com/a → example.com/b → example.com/a — you get a clean failure with the evidence attached:

RedirectLoopError: Redirect loop detected: https://example.com/a already visited

Partial hop log:
{"from": "https://example.com/a", "status": 302, "target": "https://example.com/b", "elapsed_ms": 187.2}
{"from": "https://example.com/b", "status": 302, "target": "https://example.com/a", "elapsed_ms": 165.9}

I set max_hops=10 rather than 30 because in practice, any chain longer than five hops on a scrape target means something is misconfigured — a mis-set Host header, a cookie that keeps expiring, or an A/B test bucketing you into a redirect treadmill. If you want the deeper treatment of loop causes, see Too Many Redirects: Fix Redirect Loops in Your Scraper. Ten is enough to observe the pathology without burning budget on it.

Measuring What Each Hop Actually Costs

Per-hop timing tells you two different things, and conflating them is the most common mistake I see. The elapsed time for a hop is connection setup (DNS + TCP + TLS) plus the server’s redirect decision. On the first request to a new host, connection setup dominates. On subsequent hops to the same host — thanks to connection reuse in requests.Session — the timing shrinks to almost pure server decision time.

A typical three-hop chain to store.example.com/deals, measured locally:

Step	Elapsed	Dominated by
Hop 1: example.com → example.com/deals	412 ms	Cold DNS + TLS to example.com
Hop 2: example.com/deals → store.example.com/deals	689 ms	Cold DNS + TLS to store.example.com
Hop 3: store.example.com/deals → store.example.com/deals/	204 ms	Server decision (connection reused)
Chain total	1,305 ms
Direct GET to store.example.com/deals/	587 ms	One cold connection

The chain costs 2.2x the direct request. Visualized per hop:

example.com        ████████████████████████████████████  412 ms  (cold conn)
store.example.com  ██████████████████████████████████████████████████████  689 ms  (cold conn)
store.example.com  ████████████████  204 ms  (warm conn, server think time)
                   ---------------------------------------------------------
                   Total redirect overhead: 1,305 ms
                   Direct request to final URL:  587 ms
                   Recoverable per scrape:       ~718 ms

Multiply that 718 ms across a crawl of 50,000 URLs and the redirect chain alone adds roughly ten hours of wall-clock time. That’s before considering that each hop is another opportunity for a block page dressed as a 200 — if you’re not validating intermediate responses, read Your 200 OK Responses Are Actually Block Pages and add the same validation to hop entries.

An honest caveat: the first-hop numbers are sensitive to your local DNS resolver and connection reuse. Run the tracer twice and compare — the second run’s warm-cache timings are often half the first. The number that matters for capacity planning is the cold run, because that’s what a fresh worker pod experiences.

Caching the Final Redirect Target for Faster Repeat Scrapes

Most redirect chains are stable. A legacy URL maps to the same final URL for months. Once you’ve traced it, persist the mapping and skip the chain on every subsequent scrape:

import json
import os

CACHE_FILE = "redirects.json"

def load_cache():
    if os.path.exists(CACHE_FILE):
        with open(CACHE_FILE) as f:
            return json.load(f)
    return {}

def resolve_with_cache(url):
    cache = load_cache()
    if url in cache:
        return cache[url]  # cache hit: skip the chain entirely

    hops, resp = trace_redirects_guarded(url)
    final = resp.url
    cache[url] = final
    with open(CACHE_FILE, "w") as f:
        json.dump(cache, f, indent=2)
    return final

Caches go stale, though. Sites reorganize, campaigns end, and a cached final URL starts 404ing while the original chain has been re-pointed elsewhere. Revalidate on a schedule rather than waiting for failures:

import time

def revalidate_cache(max_age_seconds=7 * 24 * 3600):
    """Re-trace cached entries weekly; log any that changed."""
    cache = load_cache()
    now = time.time()
    for source_url, entry in list(cache.items()):
        if not isinstance(entry, dict):
            cache[source_url] = {"final": entry, "checked_at": now}
            continue
        if now - entry["checked_at"] < max_age_seconds:
            continue
        hops, resp = trace_redirects_guarded(source_url)
        if resp.url != entry["final"]:
            print(f"STALE: {source_url} -> {entry['final']} "
                  f"now resolves to {resp.url}")
            cache[source_url] = {"final": resp.url, "checked_at": now}
        else:
            entry["checked_at"] = now
    with open(CACHE_FILE, "w") as f:
        json.dump(cache, f, indent=2)

My opinionated take: revalidate weekly, not daily. Daily revalidation of a large cache re-traces thousands of chains you’ll rarely need to re-trace, and the failure mode of a week-old stale entry is one bad scrape, not a data outage. Pair the cache with a fallback — if a cached URL returns a 404, re-trace the original source on the spot and update the entry.

Extending the Tracer: Headers, Cookies, and Cross-Domain Hops

Real redirect chains do things your tracer must consciously decide about. Cookies get set on hop one and silently dropped when the chain crosses to another host. Some servers redirect based on User-Agent. POST bodies vanish.

Use a requests.Session across hops so cookies carry forward, and log the cookie jar per hop so cross-domain drops are visible:

def trace_with_cookies(url, timeout=10):
    session = requests.Session()
    session.headers["User-Agent"] = "Mozilla/5.0 (compatible; myscraper/1.0)"
    hops = []
    current = url

    while True:
        resp = session.get(current, allow_redirects=False, timeout=timeout)
        if resp.status_code not in (301, 302, 303, 307, 308):
            return hops, resp
        target = urljoin(current, resp.headers.get("Location", ""))
        hops.append({
            "from": current,
            "status": resp.status_code,
            "target": target,
            "cookies_after_hop": {
                c.name: c.domain for c in session.cookies
            },
        })
        current = target

When a cookie set on example.com is absent from the jar on the store.example.com hop, the domain dump makes it obvious. If the target requires that cookie, you’ll need to re-establish a session on the final host before scraping — the chain itself can’t carry it for you.

The other decision is method and body preservation. This is where the 3xx codes genuinely differ:

Status	Method preserved	Body preserved	Practical effect
301	Rewritten to GET (de facto)	Dropped	POST becomes GET — most servers, most clients
302	Rewritten to GET (de facto)	Dropped	Same as 301 in practice
303	Always GET	Dropped	By spec, unambiguous
307	Preserved	Preserved	POST stays POST, body resent
308	Preserved	Preserved	Permanent version of 307

If you’re scraping an endpoint that expects POST — an internal search API, a filter submission — a 302 in the chain will silently convert your request to GET and you’ll get a generic listing page back with a 200 status. Nothing fails. The data is just wrong. That failure mode is why the tracer in the previous section rewrites the method explicitly and why you should log the method alongside each hop.

One more extension worth building: when the chain’s cost is structural — say the target sits behind a redirect-gating CDN and every scrape pays three hops — it’s often cheaper to hand the scrape to a managed API and let it absorb the chain cost in its own latency budget. The tracer’s value doesn’t disappear; you still want the hop log locally to attribute where the milliseconds went and to decide which URLs are worth caching.

Wrap-Up

Auto-follow redirects are convenient and opaque. Turning them off costs you one parameter and buys you per-hop status codes, per-hop timing, loop detection, and a cacheable source-to-final mapping. The pieces worth keeping from this build:

• allow_redirects=False (requests) / follow_redirects=False (httpx) — the parameter names differ, the effect is identical.
• Resolve Location with urljoin; treat a missing header as terminal.
• Log status, resolved target, and elapsed ms per hop as JSON lines — it plugs straight into any analysis you already run.
• Cap the chain at 10 hops and detect cycles with a visited set.
• Cache {source: final} mappings and revalidate weekly; fall back to a re-trace on 404.
• Watch method rewrites: 301/302/303 turn POST into GET. 307/308 don’t.

Next steps if you want to go further: extend the hop entries with response sizes and content-type so you can spot block pages mid-chain, and feed the per-hop timings into the timing harness approach from Break Down Scrape Latency: A Timing Harness You Can Run so redirect overhead shows up as its own line in your latency budget. Redirects aren’t usually the biggest cost in a scraping pipeline — but they’re the easiest one to eliminate once you can actually see them.