Keep One Exit IP Across Multi-Step Scrape Requests

Why Rotating Exit IPs Break Warmup-and-Follow-Up Scrapes

The warmup request succeeds. The next request gets redirected to the home page, receives a fresh session cookie, or returns an access challenge. Retrying makes the behavior less predictable because each attempt may arrive from another exit IP.

That pattern usually means the target bound its application session to more than the cookie. Many sites associate a session with the cookie, source IP, browser fingerprint, and request sequence. Keeping only the cookie preserves one part of that identity.

Consider this three-request sequence against https://store.example.com:

1. GET / arrives from 192.0.2.41. The server creates session sid=abc123.
2. GET /products/widget sends sid=abc123 but arrives from 198.51.100.88.
3. The server treats the IP change as a new client, invalidates or ignores sid=abc123, and returns a redirect plus a replacement cookie.

Step 1: GET https://store.example.com/
        Exit IP: 192.0.2.41
        Response: 200
        Set-Cookie: sid=abc123

Step 2: GET https://store.example.com/products/widget
        Exit IP: 198.51.100.88
        Cookie: sid=abc123
        Response: 302
        Location: https://store.example.com/
        Set-Cookie: sid=def456

Step 3: GET https://store.example.com/products/widget
        Exit IP: 203.0.113.19
        Cookie: sid=def456
        Response: session initialization starts again

Nothing is wrong with the cookie jar. The proxy rotation policy is breaking the workflow.

Step	Application session state	Cookie sent	Exit IP	Likely server response
Warmup	New session	None	192.0.2.41	200, sets sid=abc123
Follow-up after rotation	Existing session expected	sid=abc123	198.51.100.88	Redirect, challenge, or new session
Retry after another rotation	State is ambiguous	Old or replacement cookie	203.0.113.19	Another reset or inconsistent content
Follow-up without rotation	Existing session recognized	sid=abc123	192.0.2.41	Requested content

A naive fix is disabling rotation globally. That is the wrong scope. One IP should not carry every scrape indefinitely; it concentrates traffic and couples unrelated jobs. Affinity should exist only for the lifetime of one logical workflow.

Sticky routing also does not replace cookie persistence. You need both:

• A persistent HTTP client for cookies.
• One sticky identifier for exit-IP affinity.
• Consistent request headers where the target cares about client identity.
• A clean restart if either state becomes invalid.

For broader rotation design, see proxy rotation strategies.

How proxy_sticky Binds a Workflow to One Exit IP

There are two related settings, and confusing them causes bad implementations:

• proxy_sticky enables sticky routing.
• session_id identifies the workflow that should retain the same exit IP.

Do not put workflow-7f3a into proxy_sticky. That field is boolean in the API. The identifier belongs in session_id.

A two-step request flow should look like this:

Workflow: workflow-7f3a

GET https://store.example.com/
  proxy_sticky = true
  session_id    = workflow-7f3a
  exit IP       = 192.0.2.41

GET https://store.example.com/products/widget
  proxy_sticky = true
  session_id    = workflow-7f3a
  exit IP       = 192.0.2.41

An unrelated workflow gets another identifier:

Workflow: workflow-a921

GET https://store.example.com/
  proxy_sticky = true
  session_id    = workflow-a921
  exit IP       = 198.51.100.88

The exact exit addresses are illustrative. The routing property matters: equal identifiers should map to one exit during the session lifetime, while unrelated identifiers remain isolated.

Request pattern	proxy_sticky	session_id values	Expected routing
Repeated steps in one workflow	true	Same value	Same exit IP
Two unrelated workflows	true	Different values	Independently selected exits
Repeated steps without an identifier	true	Missing	Affinity scope may be unclear
Identifier changes between steps	true	Different values	Rotation is expected
Sticky routing disabled	false	Same value	No sticky guarantee

FineData exposes this distinction directly on synchronous and asynchronous scrape requests. The important design decision is still yours: define where a workflow begins, where it ends, and which requests belong to it.

Configure a Sticky Proxy Session for a Single Scrape Workflow

For a generic proxy service, the provider may encode the sticky identifier in a query parameter, username, header, or control-plane configuration. This placeholder uses https://example.com and deliberately contains no credentials:

from urllib.parse import quote

PROXY_SERVICE_URL = "https://example.com/proxy"
STICKY_ID = "workflow-7f3a"

proxy_url = (
    f"{PROXY_SERVICE_URL}"
    f"?proxy_sticky={quote(STICKY_ID, safe='')}"
)

print(proxy_url)
# https://example.com/proxy?proxy_sticky=workflow-7f3a

That URL is illustrative, not a universal proxy authentication format. Follow your provider’s documented syntax rather than assuming query parameters are supported.

With the scraping API, the equivalent configuration uses a boolean and a separate identifier:

import requests

API_URL = "https://api.finedata.ai/api/v1/scrape"
API_HEADERS = {
    "Authorization": "Bearer fd_your_api_key",
    "Content-Type": "application/json",
}

payload = {
    "url": "https://store.example.com/",
    "proxy_sticky": True,
    "session_id": "workflow-7f3a",
    "session_ttl": 1800,
    "use_residential": True,
    "formats": ["text"],
}

response = requests.post(
    API_URL,
    headers=API_HEADERS,
    json=payload,
    timeout=130,
)
response.raise_for_status()

Generate the identifier once, before creating the client or sending the warmup request:

from uuid import uuid4

def new_workflow_id() -> str:
    return f"workflow-{uuid4().hex[:12]}"

sticky_id = new_workflow_id()

# Reuse this value until the complete workflow succeeds or is abandoned.
print(sticky_id)

Keep the value in workflow state, not in a global mutable variable. A queue worker should receive it as part of the job payload. A retry handler should read the original value rather than generating another one.

I prefer opaque random identifiers over identifiers derived from customer IDs or target URLs. They avoid leaking business context into proxy logs and prevent accidental affinity between jobs that happen to target the same page.

Send the Warmup Request and Preserve Cookies

The warmup step should use the same HTTP client that will perform every follow-up request. Creating a new client per request discards the cookie jar, connection pool, and any default headers configured on the original instance.

The example below assumes a controlled store.example.com environment returns the observed source address in X-Observed-Exit-IP. Public sites generally do not expose that header, so use your own diagnostic endpoint when validating routing.

import logging
from urllib.parse import quote

import httpx

logging.basicConfig(
    level=logging.INFO,
    format="%(message)s",
)

workflow_id = "workflow-7f3a"
proxy_url = (
    "https://example.com/proxy"
    f"?proxy_sticky={quote(workflow_id, safe='')}"
)

client = httpx.Client(
    proxy=proxy_url,
    headers={
        "User-Agent": "ExampleStoreCollector/1.0",
        "Accept": "text/html,application/xhtml+xml",
    },
    follow_redirects=True,
    timeout=30.0,
)

warmup_response = client.get("https://store.example.com/")
warmup_response.raise_for_status()

warmup_exit_ip = warmup_response.headers.get(
    "X-Observed-Exit-IP",
    "unavailable",
)
warmup_cookies = dict(client.cookies.items())

logging.info(
    "workflow=%s step=warmup status=%s exit_ip=%s cookies=%s",
    workflow_id,
    warmup_response.status_code,
    warmup_exit_ip,
    sorted(warmup_cookies),
)

A safe local log contains identifiers and measurements, not proxy URLs or authentication material:

workflow=workflow-7f3a step=warmup status=200 exit_ip=192.0.2.41 cookies=['locale', 'sid']

Log cookie names if they help diagnostics. Do not log cookie values. Session cookies often grant access to server-side state and should be treated as credentials.

The warmup does not need to imitate arbitrary browsing. Request only the pages required to establish legitimate state. Extra navigation increases latency and creates more failure points without guaranteeing better acceptance.

Reuse the Same Exit IP for Follow-Up Requests

The follow-up request must reuse three things:

1. The same httpx.Client.
2. The client’s existing cookie jar.
3. The proxy configuration containing the original sticky identifier.

followup_response = client.get(
    "https://store.example.com/products/widget"
)
followup_response.raise_for_status()

followup_exit_ip = followup_response.headers.get(
    "X-Observed-Exit-IP",
    "unavailable",
)

logging.info(
    "workflow=%s step=product status=%s exit_ip=%s cookies=%s",
    workflow_id,
    followup_response.status_code,
    followup_exit_ip,
    sorted(dict(client.cookies.items())),
)

Because client is persistent, cookies set during the warmup are attached according to their domain, path, security, and expiration attributes. Do not manually build a Cookie header unless you have a specific reason; hand-built cookie strings commonly ignore path restrictions and replacement semantics.

Here is the critical comparison.

Correct: one workflow, one identifier

workflow_id = "workflow-7f3a"
proxy_url = (
    "https://example.com/proxy"
    f"?proxy_sticky={workflow_id}"
)

with httpx.Client(proxy=proxy_url, follow_redirects=True) as client:
    warmup = client.get("https://store.example.com/")
    warmup.raise_for_status()

    product = client.get(
        "https://store.example.com/products/widget"
    )
    product.raise_for_status()

Incorrect: replace the identifier between steps

warmup_proxy = (
    "https://example.com/proxy"
    "?proxy_sticky=workflow-7f3a"
)
followup_proxy = (
    "https://example.com/proxy"
    "?proxy_sticky=workflow-b819"
)

with httpx.Client(proxy=warmup_proxy) as warmup_client:
    warmup = warmup_client.get("https://store.example.com/")
    warmup.raise_for_status()
    cookies = dict(warmup_client.cookies.items())

with httpx.Client(
    proxy=followup_proxy,
    cookies=cookies,
) as followup_client:
    product = followup_client.get(
        "https://store.example.com/products/widget"
    )

The incorrect version copies cookies but changes the exit identity. It may appear functional against a permissive target, which makes the defect easy to miss during development.

A sticky IP is not automatically desirable for an entire crawl. Long-lived affinity reduces rotation and can concentrate hundreds of requests on one address. Scope it to the smallest sequence that genuinely requires continuity, then close the client.

Verify IP Affinity and Detect Unexpected Rotation

Do not infer affinity from successful responses. A target may tolerate rotation for two steps and reject it later. Measure the exit address through infrastructure you control.

For example, configure these controlled routes to return the caller’s source address in X-Observed-Exit-IP:

paths = [
    "https://store.example.com/",
    "https://store.example.com/products/widget",
    "https://store.example.com/products/widget/reviews",
]

observed_exit_ips = []

for step, url in enumerate(paths, start=1):
    response = client.get(url)
    response.raise_for_status()

    exit_ip = response.headers["X-Observed-Exit-IP"]
    observed_exit_ips.append(exit_ip)

    logging.info(
        "workflow=%s step=%s status=%s exit_ip=%s",
        workflow_id,
        step,
        response.status_code,
        exit_ip,
    )

assert len(set(observed_exit_ips)) == 1, (
    f"exit IP rotated inside workflow: {observed_exit_ips}"
)

Expected local output:

workflow=workflow-7f3a step=1 status=200 exit_ip=192.0.2.41
workflow=workflow-7f3a step=2 status=200 exit_ip=192.0.2.41
workflow=workflow-7f3a step=3 status=200 exit_ip=192.0.2.41

In production, stop before sending another state-dependent request when a mismatch appears:

class ExitIpChanged(RuntimeError):
    pass

def require_same_exit_ip(
    workflow_id: str,
    expected_ip: str,
    response: httpx.Response,
) -> None:
    actual_ip = response.headers.get("X-Observed-Exit-IP")

    if not actual_ip:
        raise ExitIpChanged(
            f"workflow={workflow_id} cannot verify exit IP"
        )

    if actual_ip != expected_ip:
        logging.error(
            "workflow=%s error=exit_ip_changed expected=%s actual=%s",
            workflow_id,
            expected_ip,
            actual_ip,
        )
        raise ExitIpChanged(
            f"workflow={workflow_id} exit IP changed "
            f"from {expected_ip} to {actual_ip}"
        )

followup = client.get(
    "https://store.example.com/products/widget"
)
require_same_exit_ip(
    workflow_id,
    warmup_exit_ip,
    followup,
)

Failing closed is the better policy here. Continuing with a mismatched address can corrupt application state, replace valid cookies, and make the eventual error harder to diagnose.

Handle Retries, Parallel Jobs, and Sticky Session Expiration

A retry of one failed step is still part of the original workflow. Reuse its identifier. A restart from the warmup step is a new workflow and should receive a new identifier, a fresh cookie jar, and a newly selected exit.

The synchronous API supports automatic retries while retaining the request’s session configuration:

from uuid import uuid4

import requests

API_URL = "https://api.finedata.ai/api/v1/scrape"
HEADERS = {
    "Authorization": "Bearer fd_your_api_key",
    "Content-Type": "application/json",
}

def workflow_payload(session_id: str) -> dict:
    return {
        "url": "https://store.example.com/products/widget",
        "proxy_sticky": True,
        "session_id": session_id,
        "session_ttl": 1800,
        "use_residential": True,
        "auto_retry": True,
        "max_retries": 3,
        "timeout": 120,
        "formats": ["text"],
    }

# Transient retry configuration preserves this identifier.
current_session_id = "workflow-7f3a"
response = requests.post(
    API_URL,
    headers=HEADERS,
    json=workflow_payload(current_session_id),
    timeout=130,
)

# A complete restart gets a new identifier.
if not response.ok:
    restarted_session_id = f"workflow-{uuid4().hex[:12]}"
    restarted_payload = workflow_payload(restarted_session_id)

Do not retry indefinitely. Three retries are enough to distinguish a brief transport problem from a broken workflow in many pipelines. More attempts can spend quota while repeatedly exercising invalid state.

Situation	Reuse sticky identifier?	Reuse cookies?	Action
Transient retry of the same step	Yes	Yes	Retry through the existing client
Next step in the same workflow	Yes	Yes	Continue after verifying affinity
Parallel scrape job	No	No	Create isolated identifier and client
Sticky session expired	No	No	Restart from warmup
Exit IP mismatch detected	No	No	Abort and create a fresh workflow
Target explicitly replaces a valid cookie	Usually yes	Let the jar update	Continue only if the exit IP is unchanged

Parallelism needs isolation, not shared affinity. Two jobs can target the same domain while maintaining independent cookies and exit addresses:

from concurrent.futures import ThreadPoolExecutor
from urllib.parse import quote

import httpx

def run_job(workflow_id: str) -> tuple[str, list[str]]:
    proxy_url = (
        "https://example.com/proxy"
        f"?proxy_sticky={quote(workflow_id, safe='')}"
    )

    urls = [
        "https://store.example.com/",
        "https://store.example.com/products/widget",
    ]

    with httpx.Client(
        proxy=proxy_url,
        follow_redirects=True,
        timeout=30.0,
    ) as client:
        exit_ips = []

        for url in urls:
            response = client.get(url)
            response.raise_for_status()
            exit_ips.append(
                response.headers["X-Observed-Exit-IP"]
            )

    if len(set(exit_ips)) != 1:
        raise RuntimeError(
            f"workflow={workflow_id} rotated: {exit_ips}"
        )

    return workflow_id, exit_ips

with ThreadPoolExecutor(max_workers=2) as executor:
    results = list(
        executor.map(
            run_job,
            ["workflow-7f3a", "workflow-a921"],
        )
    )

for workflow_id, exit_ips in results:
    print(workflow_id, exit_ips)

Possible output:

workflow-7f3a ['192.0.2.41', '192.0.2.41']
workflow-a921 ['198.51.100.88', '198.51.100.88']

The two jobs do not need different exit addresses by definition; a proxy pool may select the same address for both. What matters is that each job remains internally consistent and never shares cookies with the other.

Set the session lifetime longer than the expected workflow duration, including queue delay and retries. Do not stretch it far beyond that window. If a workflow normally finishes within ten minutes, a thirty-minute lifetime leaves practical retry room without turning affinity into a permanent assignment.

For workloads that outlive one HTTP request, asynchronous jobs and callbacks may be a cleaner execution model. The same isolation rules still apply; see async scraping jobs and webhooks.

Wrap-up

A multi-step scrape has two kinds of state: target state in the cookie jar and network identity in the proxy layer. Preserve both for the workflow’s lifetime.

Enable proxy_sticky, assign one session_id, reuse the same HTTP client, and verify the observed address before continuing. Retries keep the identifier. Parallel jobs get separate identifiers. Expired or rotated sessions restart from warmup with fresh cookies. That boundary keeps session continuity predictable without disabling proxy rotation across the rest of the pipeline.