Sticky Exit IPs for Warmup and Follow-Up Scrapes

Same Cookie Jar, Different Exit: Why the Product GET Returns 403

A warmup GET to https://store.example.com/ returns 200 and two Set-Cookie headers. The follow-up GET to https://store.example.com/products/sku-1001, issued 1.8s later from the same process, returns 403. The jar still holds sid and locale. The exit IP does not. Hop 1 left through 203.0.113.10. Hop 2 left through 198.51.100.44. The origin treated the second request as a new visitor presenting a cookie it did not mint for that address.

Per-request rotation is the right default for independent page fetches. It is the wrong default for a warmup-then-read pair. Retrying the product URL does not fix it. You are still rolling the pool. Stripping cookies “to look less like a replay” avoids the 403 and also throws away the locale, tax, and catalog the warmup existed to create. Pin the exit across both hits, or stop pretending this is one session.

Rotating Exits Between Warmup and Follow-Up Drop Set-Cookie and Geo Affinity

Status codes hide the split. Trace both hops against the same host.

GET / HTTP/1.1
Host: store.example.com
User-Agent: Mozilla/5.0
Accept-Language: de-DE,de;q=0.9

HTTP/1.1 200 OK
Set-Cookie: sid=a8f3c1e90b; Domain=.example.com; Path=/; Secure; HttpOnly; SameSite=Lax
Set-Cookie: locale=de-DE; Domain=.example.com; Path=/; Secure; SameSite=Lax
X-Egress-IP: 203.0.113.10
X-Client-Geo: DE-Berlin
Content-Type: text/html; charset=utf-8

Think-time: 1840 ms. Same process, same jar, next request:

GET /products/sku-1001 HTTP/1.1
Host: store.example.com
User-Agent: Mozilla/5.0
Accept-Language: de-DE,de;q=0.9
Cookie: sid=a8f3c1e90b; locale=de-DE

HTTP/1.1 403 Forbidden
X-Egress-IP: 198.51.100.44
X-Client-Geo: US-Dallas
X-Deny: session_ip_mismatch
Content-Type: application/json

Set-Cookie on hop 1 did its job. The jar replayed sid and locale verbatim. What changed is the address the store actually saw. Fraud and geo middleware bind that cookie to the first-seen exit — IP, /24, ASN, or city, depending on the vendor. A Dallas egress presenting a Berlin session looks like cookie theft, not a shopper.

Sometimes you do not even get 403. You get 200 with a USD price list, an empty size run, or a tax-inclusive flag that does not match locale=de-DE. That is worse than a hard fail: your pipeline stores a page that is internally consistent and commercially false.

A second product GET through yet another exit does not converge. Each rotate is a new first-seen address. The cookie still says Berlin. The packet still says somewhere else. Until both hops share an exit, you are measuring the pool, not the store.

How store.example.com Binds Session Cookies and Locale to the First-Seen Exit

Read the cookie line. Then assume it is not the only check.

Set-Cookie: sid=a8f3c1e90b; Domain=.example.com; Path=/; Secure; HttpOnly; SameSite=Lax
# Domain=.example.com  -> every host under example.com, not host-only
# Path=/               -> all paths, including /products/sku-1001
# Secure               -> dropped on plaintext http://
# HttpOnly             -> JS cannot read it; your HTTP client still can
# SameSite=Lax         -> sent on top-level GETs; irrelevant for server-side scrapes

locale=de-DE usually rides the same attributes. Browsers honor this. Origins often add a second binding the RFC does not describe: the exit they observed when they issued sid.

Binding	What must match on hop 2	Typical failure
Cookie jar only	Cookie header	Rare on storefronts with fraud scoring
Cookie + country	Jar + ISO country of the exit	DE session + US egress → 403 or USD catalog
Cookie + city / ASN	Jar + city or ASN of hop 1	Two DE residentials, Berlin vs Frankfurt, still denied

Cookie-jar-only stores are the ones people remember from tutorials. They are not the ones that burn a day of logs. Country binding is the usual storefront case: licensed catalog, VAT, shipping. City/ASN binding shows up when a risk engine sits in front of /products/* and treats ASN hops as account takeover.

Do not guess the row. Compare X-Egress-IP / X-Client-Geo (or whatever echo your gateway injects) on both responses. Country matches and you still get session_ip_mismatch → you need IP persistence, not just proxy_country=DE. Country mismatches → a country pin may be enough, and a sticky IP is wasted hold time on a residential port.

I would rather start at country pinning and upgrade after a traced 403 than hold an exit “just in case.” Warmup hits get cargo-culted. If hop 2 succeeds with a rotated DE address and the same jar, the store is not binding IP. Skip the pin. Sticky ports cost more and attract reputation heat when you sit on them.

Cookie attributes still matter for scope. Domain=.example.com follows you to store.example.com and to cdn.example.com. A host-only cookie (no Domain attribute) stays on the exact host that set it. Both URLs in this flow are store.example.com, so host-only survives. The moment a later hop moves to another hostname under the same registrable domain, host-only dies even if the exit IP did not move. That is a cookie-scope bug, not a proxy bug. Fix the jar rules; do not lengthen TTL.

TLS is a separate axis. Two hops on one IP with two client hellos still look like two tools. Pinning the exit does not flatten JA3. Keep one HTTP client for the pair, and read TLS fingerprinting before you pile on browser flags you do not need.

Sticky Exit IPs: One Residential PoP for Warmup Plus the Immediate Follow-Up

A sticky session is a pool instruction: given a session key, keep returning the same exit IP until TTL fires or the provider force-rotates. Per-request rotation is the opposite instruction: ignore affinity, pick the next warm port.

The unit of stickiness for this problem is the pair — homepage then product — not the crawl.

	Rotating (control)	Sticky (treatment)
IP equality hop 1 vs hop 2	No	Yes, until TTL or force-rotate
Cookie survival	Header is sent; origin may reject	Header is sent and origin can accept
Reported city	Can jump between hops	Matches hop 1
Minimum TTL	None (and that is the bug)	Greater than think-time; 1800s covers a warmup-then-scrape gap with margin

TTL must outlive the gap, not match it. A 2s think-time with a 2s TTL loses the race against reaper clocks. The scrape API default session_ttl of 1800 seconds is already larger than any honest warmup-to-product pause. The max of 86400 seconds is a foot-gun: one exit held all day becomes the identity of every SKU you touch.

Here is the part many pipelines get backwards. They mint one session key for a 10k URL job and call it “stability.” It is concentration. When that IP is blacklisted, the whole job dies, and you taught the origin a durable fingerprint. Use stickiness for the pair, then drop it. Independent product URLs should go back to rotation. Proxy rotation strategies exist for the bulk path; this article is about the exception.

Residential vs datacenter: if the store keys on ASN, a sticky datacenter IP is a sticky bad ASN. use_residential=true is the usual match for cookie+city/ASN bindings. It is also slower and priced higher. Do not turn it on because the flag exists. Turn it on because the trace showed a residential ASN on the browser path you are imitating.

proxy_sticky on a single scrape job keeps the exit for that browser session (useful when js_actions click through a warmup inside one job). Two separate HTTP calls from your process need a session identifier that outlives a single response: session_id on the API, or sessid= on the proxy URL. Those are not interchangeable names for the same knob. One pins inside a job. The other pins across jobs.

Proxy URL and Session Key That Outlive the Warmup-to-Scrape Gap

The session key must be identical on both hops. The TTL must be an explicit number you chose, not “whatever the provider defaulted to in a dashboard you have not opened.”

# Pair:
#   GET https://store.example.com/
#   GET https://store.example.com/products/sku-1001
# sessid=warmup-42  — stable key; both hops MUST send this exact value
# ttl=1800          — seconds the pool should hold the exit (default 1800, max 86400)
#                     1800s >> think-time; do not set 86400 for a two-hit flow
PROXY_URL=http://user:pass@203.0.113.8:8000?sessid=warmup-42&ttl=1800

No second hostname. No second sessid. If hop 2 rebuilds the URL and omits sessid, you silently rotated. That bug looks like “sticky does not work.”

Dispatching each hop as its own scrape job needs the same idea in JSON: reuse session_id, set session_ttl past the gap, include url on every body, and ask for a residential exit if the trace said you need one. Illustration only — not executed:

{"url": "https://example.com", "method": "GET", "use_residential": true, "proxy_sticky": true, "session_id": "warmup-42", "session_ttl": 1800}

session_id pins the exit across those two jobs. It does not pin your cookie jar. The jar lives in your process unless you copy Cookie on the second job yourself. Two API jobs with a shared session_id and no cookie plumbing is how people “enable sticky” and still 403. I prefer a local client for this pair: one jar, one proxy URL, one TLS stack, cheaper than two full scrape jobs for a homepage plus a product page.

import httpx

PROXY_URL = "http://user:pass@203.0.113.8:8000?sessid=warmup-42&ttl=1800"

with httpx.Client(proxy=PROXY_URL, timeout=120.0, follow_redirects=True) as client:
    warmup = client.get("https://example.com")
    follow_up = client.get("https://example.com")

session_id max length is 64. warmup-42 is fine. Do not hash a paragraph of metadata into the key. Do not reuse warmup-42 tomorrow on a different machine unless you intend to share the exit.

Python Client: Warmup GET then Product GET on One Sticky Session and Cookie Jar

One httpx.Client owns the jar, the proxy URL, and the connection behavior. Two ad-hoc httpx.get(...) calls with a proxies= dict copied from a gist will desync at least one of those.

import time
import httpx

PROXY_URL = "http://user:pass@203.0.113.8:8000?sessid=warmup-42&ttl=1800"

def dump(label: str, response: httpx.Response, t0: float) -> None:
    print(label)
    print("  status    ", response.status_code)
    print("  egress    ", response.headers.get("x-egress-ip", "missing"))
    print("  geo       ", response.headers.get("x-client-geo", "missing"))
    print("  set-cookie", response.headers.get_list("set-cookie"))
    print("  elapsed_ms", round((time.monotonic() - t0) * 1000, 1))

timeout = httpx.Timeout(120.0)
with httpx.Client(
    proxy=PROXY_URL,
    timeout=timeout,
    follow_redirects=True,
    headers={
        "User-Agent": "Mozilla/5.0",
        "Accept-Language": "de-DE,de;q=0.9",
    },
) as client:
    t_home = time.monotonic()
    home = client.get("https://store.example.com/")
    dump("warmup", home, t_home)
    home.raise_for_status()

    t_product = time.monotonic()
    product = client.get("https://store.example.com/products/sku-1001")
    dump("follow-up", product, t_product)

    print("jar", list(client.cookies.jar))
    print("gap_ms", round((t_product - t_home) * 1000, 1))
    print("sid_replayed", "sid" in client.cookies)

client.cookies is the jar. Hop 2 sends whatever hop 1 stored. sessid=warmup-42 is how the pool keeps X-Egress-IP still. If you construct a second Client for the product GET — even with the same PROXY_URL — you still share the exit, but you drop the jar unless you pass it across. That is a frequent copy-paste fail: sticky IP, empty Cookie, origin mints a second sid, and now you have IP affinity with session confusion.

I prefer httpx over requests here because proxy=, cookies, and timeout sit on one Client. requests.Session can do it, but the proxies= dict vs environment vs trust-env mix is how people rotate without meaning to.

follow_redirects=True is required if / 302s to /de/ and sets cookies on the redirect hop. Disable it and you will log a 200 on a URL you never warmed.

Print x-egress-ip from the gateway. If the header is missing, you are not measuring. Do not infer equality from “both returned 200.”

Local Measurement: Exit IP Equality, Cookie Reuse, and Gap Latency

Run the client twice: once with sessid removed (control), once with sessid=warmup-42 (treatment). Record egress, cookie presence on the wire, status, and gap.

# rotating-control  (no sessid; pool picks per request)
warmup
  status     200
  egress     203.0.113.10
  geo        DE-Berlin
  set-cookie ['sid=a8f3c1e90b; Domain=.example.com; Path=/; Secure; HttpOnly; SameSite=Lax',
              'locale=de-DE; Domain=.example.com; Path=/; Secure; SameSite=Lax']
  elapsed_ms 942.0
follow-up
  status     403
  egress     198.51.100.44
  geo        US-Dallas
  set-cookie []
  elapsed_ms 1104.2
jar [Cookie(sid=a8f3c1e90b), Cookie(locale=de-DE)]
gap_ms 1842.6
sid_replayed True
ip_equal False

# sticky-treatment  (sessid=warmup-42&ttl=1800)
warmup
  status     200
  egress     203.0.113.10
  geo        DE-Berlin
  set-cookie ['sid=a8f3c1e90b; Domain=.example.com; Path=/; Secure; HttpOnly; SameSite=Lax',
              'locale=de-DE; Domain=.example.com; Path=/; Secure; SameSite=Lax']
  elapsed_ms 1003.4
follow-up
  status     200
  egress     203.0.113.10
  geo        DE-Berlin
  set-cookie []
  elapsed_ms 881.7
jar [Cookie(sid=a8f3c1e90b), Cookie(locale=de-DE)]
gap_ms 1889.1
sid_replayed True
ip_equal True

Run	ip_equal	sid on request 2	gap_ms	Product status
rotating-control	no (203.0.113.10 vs 198.51.100.44)	yes	1842.6	403
sticky-treatment	yes (203.0.113.10)	yes	1889.1	200

Cookie reuse without IP equality is the control failure mode. IP equality without the cookie is a Client bug, not a pool bug.

To find how long you can wait, keep sessid fixed and increase the sleep between hops: 5s, 60s, 600s, 1700s, 1900s. When ip_equal flips to false, you found the real hold time. Believe that number, not the brochure. If equality dies at 1700s with ttl=1800, your reaper is early; set 2400 and stop there. If equality still holds at 1900s, do not “use the slack” to batch fifty SKUs onto that exit.

gap_ms in the 1–2s range is think-time, not a durability test. A pass at 1889 ms only proves the pin survived the pair. It does not prove a 20-minute human browse.

When Sticky Still Fails: TTL Expiry, Forced Rotation, and .example.com Cookie Scope

Pins die. Providers force-rotate under load, health checks, or target blocks. TTL hits zero while your worker sat on a queue. Domain=.example.com cookies leak onto a host you did not warm; host-only cookies refuse to follow a subdomain you did. None of those are fixed by retrying /products/sku-1001 on the same jar.

Treat IP change or 403 on store.example.com as session death. Drop the jar. Mint a new sessid. Warm up again. Then follow up. Mixing a Berlin sid onto a fresh Dallas exit is how you train the deny list.

import secrets
import time
import httpx

GATEWAY = "http://user:pass@203.0.113.8:8000"
TIMEOUT = httpx.Timeout(120.0)

def proxy_url(sessid: str, ttl: int = 1800) -> str:
    return f"{GATEWAY}?sessid={sessid}&ttl={ttl}"

def session_dead(home: httpx.Response, product: httpx.Response) -> bool:
    ip1 = home.headers.get("x-egress-ip")
    ip2 = product.headers.get("x-egress-ip")
    if not ip1 or not ip2 or ip1 != ip2:
        return True
    if product.status_code in {401, 403, 429}:
        return True
    return False

def warmup_then_product(max_attempts: int = 3) -> httpx.Response:
    for _ in range(max_attempts):
        sessid = f"warmup-{secrets.token_hex(4)}"  # new key, never reuse a dead one
        with httpx.Client(
            proxy=proxy_url(sessid),
            timeout=TIMEOUT,
            follow_redirects=True,
            headers={"User-Agent": "Mozilla/5.0", "Accept-Language": "de-DE,de;q=0.9"},
        ) as client:
            home = client.get("https://store.example.com/")
            if home.status_code != 200 or "sid" not in client.cookies:
                client.cookies.clear()
                continue
            # host-only vs Domain=.example.com:
            # both hops share store.example.com, so either form is sent.
            # if a future hop moved host, filter cookies whose domain does not cover it.
            product = client.get("https://store.example.com/products/sku-1001")
            if session_dead(home, product):
                client.cookies.clear()  # drop sid/locale; do not replay on the next exit
                time.sleep(0.5)
                continue
            return product
    raise RuntimeError("sticky pair failed after fresh sessid retries")

Rules that matter in that path:

• New sessid every resurrection. Reusing warmup-42 after a force-rotate can pin you to the burned IP, or to nothing, depending on provider semantics. Do not find out in production.
• client.cookies.clear() before the next attempt. Dead session plus live cookie is the original 403.
• Do not special-case 429 as “slow down and replay.” Back off, then start at /, not at /products/sku-1001.
• Domain=.example.com is sent to every matching host. Host-only (no Domain) is not. If you ever add a hop to another hostname, split the jar; a sticky IP will not carry a host-only sid across hosts.
• Cap attempts. Three full warmup+product pairs is plenty. A loop that mints sessids forever is a self-inflicted block.

Forced rotation mid-pair will still happen at scale. The fix is fail-closed, not “try the product URL three more times.” I will take a longer wall clock over a polluted cookie that poisons the next hundred SKUs.

Pin the exit for the pair, not for the crawl. Trace hop 1 against hop 2 until X-Egress-IP matches and sid is actually replayed. If country already matches and the product GET is 200, you do not need sticky IP. If the IP flips and the store 403s, one sessid, one jar, one TTL past think-time, and a hard reset when any of those break. That is the whole mechanism.