Tutorial 14 min read

Route Scraping Traffic Through Proxies You Already Own

When targets allowlist your IPs or you already pay for residential proxies, attach a proxy profile so scrape requests exit through your pool.

FE
FineData Engineering · Editorial Policy
| | Updated September 6, 2026

Named proxy profiles send catalog traffic through IPs store.example.com already allowlists

The catalog job is green in your runner and still comes back 403. store.example.com already has your office /24 in its WAF allowlist. You prepaid a residential pool last quarter. None of that matters if the scraper leaves from a vendor IP the target has never seen.

Your allowlist is doing its job. The scraper is not using it. A one-off curl --proxy against the catalog looks fine from a laptop, then the scheduled job repeats the 403 because egress never left your network. Pasting host:port into every job file is the other failure mode: credentials leak into git, rotation is handwritten, and a single dead endpoint kills the crawl. A named proxy profile is the fix — register the endpoints you already pay for, attach that profile to the job, and keep secrets out of the job file.

Scrape jobs still leave from vendor IPs after store.example.com allowlists yours

The symptom is boring and consistent. You open a ticket with the store’s infra team. They add 203.0.113.0/24. You rerun the job. They still see a request from 198.51.100.44. Their WAF does exactly what you asked: reject anything outside the allowlist.

Here is the path that actually fired versus the path you thought you configured:

RequestSource IPHTTP status
GET https://store.example.com/catalog via default scraper egress198.51.100.44 (vendor pool)403
GET https://store.example.com/catalog via owned datacenter proxy203.0.113.17 (allowlisted /24)200

The 403 is not a selector bug. It is an identity mismatch. A failed transcript looks like this:

$ curl -sI https://store.example.com/catalog
HTTP/2 403
content-type: text/html; charset=utf-8
cache-control: no-store
x-deny-reason: ip-not-allowlisted

<!DOCTYPE html>
<html>
  <head><title>Access denied</title></head>
  <body>
    <p>Origin IP is not on the store.example.com allowlist.</p>
  </body>
</html>

Three naive fixes show up in every postmortem. Allowlisting the vendor’s published ranges fails the moment they rotate an exit. Setting HTTPS_PROXY on the worker host also wraps health checks, metadata fetches, and webhook delivery — those calls now hairpin through a residential endpoint that has no business seeing your internal traffic. Hand-editing a proxy URL into one scrape body works once, then the next job ships without it.

If the target already trusts your IPs, the scraper has to leave through those IPs. Everything else is theatre.

Register datacenter and residential endpoints you already pay for as one named profile

Stop pasting endpoints into jobs. Register each pool you already pay for — datacenter or residential — under its own name, and treat that name as the unit of rotation. A profile is a list of proxy URLs plus a rotation mode, and the same register shape applies to both. The members must not be mixed.

Each proxy is a URL string — scheme://[user:pass@]host:port — not a nested object. The accepted schemes are http, https, socks5, and socks5h. Most authenticated forward proxies use http. Use the SOCKS variants only if the vendor documents CONNECT support on that port; otherwise you will debug handshake failures that look like target 403s.

Keep a YAML source of truth in git with placeholders only. Canonical catalog profile: two HTTP datacenter exits in the allowlisted /24. No residential member. The credentials are environment placeholders, never real passwords.

name: store-allowlist-dc
rotation_mode: round_robin
auto_skip_failed: true
proxies:
  - "http://dc_user_a:${PROXY_PASS_DC}@dc1.proxy.example.com:8080"
  - "http://dc_user_b:${PROXY_PASS_DC}@dc2.proxy.example.com:8080"

Map each URL back to the provider dashboard before you register anything. host and port are the “endpoint” or “gateway” row, not the customer portal hostname. username is rarely a bare login. Datacenter accounts are often a static user per box (dc_user_a, dc_user_b). Residential accounts encode routing in the user string (user-country-us, sometimes user-country-us-session-ABCD) and belong in a different profile. The password segment is the token in the “authentication” panel, not your account password.

Create that DC profile from those values. Base URL, API key, and proxy passwords all come from the environment, never from the YAML committed to the repo. Set SCRAPE_API to the control plane base URL.

import os
import requests

API = os.environ["SCRAPE_API"]
headers = {
    "Authorization": f"Bearer {os.environ['SCRAPE_API_KEY']}",
    "Content-Type": "application/json",
}

dc_pass = os.environ["PROXY_PASS_DC"]

resp = requests.post(
    f"{API}/api/v1/proxy-profiles",
    headers=headers,
    json={
        "name": "store-allowlist-dc",
        "rotation_mode": "round_robin",
        "auto_skip_failed": True,
        "proxies": [
            f"http://dc_user_a:{dc_pass}@dc1.proxy.example.com:8080",
            f"http://dc_user_b:{dc_pass}@dc2.proxy.example.com:8080",
        ],
    },
)
resp.raise_for_status()
print(resp.json())  # persist the integer profile_id; export it as PROXY_PROFILE_ID

Do not dump datacenter allowlist IPs and rotating residential exits into the same profile. Catalog traffic that must land in 203.0.113.0/24 cannot afford a round-robin hop onto a residential IP the WAF has never seen. Split the pools: store-allowlist-dc for the trusted /24, a second profile (next section) for targets that do not know you. Mixing them is convenient and wrong. Plenty of teams still mix them; they then “debug” intermittent 403s that are just the residential member of the pool.

List what you registered, then read one profile if the name is ambiguous. Reuse API and headers from the create call:

listed = requests.get(f"{API}/api/v1/proxy-profiles", headers=headers)
listed.raise_for_status()
print(listed.json())

profile_id = 12  # the integer returned at create time
one = requests.get(f"{API}/api/v1/proxy-profiles/{profile_id}", headers=headers)
one.raise_for_status()
print(one.json())

Attach the profile so every request to store.example.com exits through your pool

A profile that is not referenced by the job is dead config. The scrape body needs proxy_profile_id. That single field is what moves egress onto your pool. Default routing, the vendor proxy flags (use_residential, use_isp, use_mobile), and a BYOP profile are mutually exclusive — pick the profile and do not also flip vendor flags on the same request.

This is the canonical catalog submit. Later sections only add fields; they do not restate this POST.

import os
import requests

API = os.environ["SCRAPE_API"]
headers = {
    "Authorization": f"Bearer {os.environ['SCRAPE_API_KEY']}",
    "Content-Type": "application/json",
}

profile_id = int(os.environ["PROXY_PROFILE_ID"])

scrape_body = {
    "url": "https://store.example.com/catalog",
    "method": "GET",
    "formats": ["text"],
    "proxy_profile_id": profile_id,
    "callback_url": "https://api.example.com/hooks/scrape",
}

job = requests.post(f"{API}/api/v1/async/scrape", headers=headers, json=scrape_body)
job.raise_for_status()
print(job.json())

The same change as a job-file diff:

  url: https://store.example.com/catalog
  method: GET
  formats:
    - text
+ proxy_profile_id: 12

Before, the runner posted {url, method, formats} and left through vendor IPs. After, every request in that job selects an endpoint from store-allowlist-dc under round_robin. Updating an existing profile (new datacenter box) is a PUT on the profile, not a rewrite of every job:

requests.put(
    f"{API}/api/v1/proxy-profiles/{profile_id}",
    headers=headers,
    json={
        "name": "store-allowlist-dc",
        "rotation_mode": "round_robin",
        "auto_skip_failed": True,
        "proxies": [
            f"http://dc_user_a:{os.environ['PROXY_PASS_DC']}@dc1.proxy.example.com:8080",
            f"http://dc_user_b:{os.environ['PROXY_PASS_DC']}@dc3.proxy.example.com:8080",
        ],
    },
).raise_for_status()

Jobs keep proxy_profile_id: 12. The pool behind that id changes. That is the point of naming it.

Synchronous POST /api/v1/scrape takes the same scrape_body (drop callback_url, add timeout if you want a client-side cap). Use async when the catalog crawl is long enough that holding the HTTP client open is the bottleneck, not the proxy.

Keep rotating residential user:pass in secrets, not in the job file

Residential vendors rotate credentials, encode country in the username, and expire session suffixes. None of that belongs in a job YAML that ten people can clone, and none of it belongs in store-allowlist-dc. Same register schema, different name, different pool.

Export the pair in the runner environment. Static DC usernames stay in the profile YAML; rotating residential usernames do not.

export SCRAPE_API
export SCRAPE_API_KEY
export PROXY_USER="user-country-us"
export PROXY_PASS
export PROXY_PROFILE_ID
export PROXY_PASS_DC

Read credentials at runtime, then create or refresh the residential profile. Node is enough for a sidecar that never writes the password to disk:

const api = process.env.SCRAPE_API;
const key = process.env.SCRAPE_API_KEY;
const user = process.env.PROXY_USER;
const pass = process.env.PROXY_PASS;
if (!api || !key || !user || !pass) {
  throw new Error("SCRAPE_API, SCRAPE_API_KEY, PROXY_USER, and PROXY_PASS must be set");
}

const res = await fetch(`${api}/api/v1/proxy-profiles`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${key}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "res-rotating",
    rotation_mode: "round_robin",
    auto_skip_failed: true,
    proxies: [
      `http://${encodeURIComponent(user)}:${encodeURIComponent(pass)}@res.proxy.example.com:60000`,
    ],
  }),
});
if (!res.ok) {
  throw new Error(`profile create failed: ${res.status}`);
}
console.log(await res.json());

The job file that points at store.example.com should mention the profile id and nothing that looks like a password. For the allowlisted catalog, that id is store-allowlist-dc (12), not res-rotating.

url: https://store.example.com/catalog
method: GET
formats:
  - text
proxy_profile_id: 12
# PROXY_USER / PROXY_PASS hydrate res-rotating in the sidecar, not this file
# never inline: http://user:pass@res.proxy.example.com:60000

If a username carries a session suffix (user-country-us-session-ABCD), generate that suffix in the same sidecar and PUT the residential profile. Do not commit the suffix. Do not log the constructed URL. encodeURIComponent on user and password is not optional; residential passwords are often token-shaped and will break URL parsing on the first + or /.

The control plane stores the pool on the profile, so the job YAML stays boring. Boring is what you want in git.

Prove egress IP matches the allowlist before you scale the crawl

Do not scale on a green runner checkbox. Measure the public IP the DC proxy actually uses, then hit the catalog with store-allowlist-dc. Hardcoding an IP in a script is how you ship last week’s exit.

Probe an echo endpoint through one owned DC proxy — not store.example.com. A catalog 403 is an allowlist miss; a proxy 407/502 is a dead member. Those must not share a code path.

# Local check: parse whatever the echo body returns. Do not paste an IP into the script.
curl -sS \
  --proxy "http://dc_user_a:${PROXY_PASS_DC}@dc1.proxy.example.com:8080" \
  https://api.example.com/ip

import ipaddress
import json
import os
import subprocess

raw = subprocess.check_output(
    [
        "curl", "-sS",
        "--proxy",
        f"http://dc_user_a:{os.environ['PROXY_PASS_DC']}@dc1.proxy.example.com:8080",
        "https://api.example.com/ip",
    ],
    text=True,
)
echo = json.loads(raw)
egress_ip = echo["ip"]  # field name follows the echo JSON; read it, do not hardcode
print(f"proxy_egress={egress_ip}")

allowlist = ipaddress.ip_network("203.0.113.0/24")
if ipaddress.ip_address(egress_ip) not in allowlist:
    raise SystemExit(f"{egress_ip} is outside {allowlist}")

When that assert passes, submit the catalog URL with the scrape_body from the previous section (proxy_profile_id already set). Do not paste the egress IP into the scrape body.

Run the profile test endpoint as a second signal that the pool is reachable from the API side:

import os
import requests

API = os.environ["SCRAPE_API"]
headers = {
    "Authorization": f"Bearer {os.environ['SCRAPE_API_KEY']}",
    "Content-Type": "application/json",
}
profile_id = int(os.environ["PROXY_PROFILE_ID"])

test = requests.post(
    f"{API}/api/v1/proxy-profiles/{profile_id}/test",
    headers=headers,
)
test.raise_for_status()
print(test.json())

Checklist before you widen concurrency:

  • Echo JSON parsed, no IP literals in the script
  • Parsed egress address sits inside the CIDR you actually allowlisted (203.0.113.0/24 in this example)
  • store-allowlist-dc lists only those DC exits — no residential member in the profile
  • GET https://store.example.com/catalog through proxy_profile_id is not a 403
  • Profile test call returns without an error payload

If the echo IP is inside the CIDR and the catalog still 403s, the allowlist is on the wrong layer (CDN vs origin) or bound to a header the WAF expects. That is a target-config problem. Fixing it by disabling the profile puts you back on vendor IPs.

Pin a sticky proxy when store.example.com binds cookies and carts to IP

round_robin is the right default for /catalog. It is the wrong default for /cart on a store that keys the session to exit IP plus cookie. Rotate mid-session and you get a new cart, a dropped cookie, or a silent redirect to an empty basket. Prefer request-level stickiness over rotation_mode: sticky on the whole profile, because catalog workers and cart workers should not share that setting.

Contrast the two configs for https://store.example.com/cart. Same proxy_profile_id as the catalog job; only the sticky fields change:

# Rotate every request — breaks IP-bound cookies
url: https://store.example.com/cart
proxy_profile_id: 12
# rotation_mode on the profile is round_robin
# no session_id, proxy_sticky remains false

# One exit for this browser session, rotate across sessions
url: https://store.example.com/cart
proxy_profile_id: 12
proxy_sticky: true
session_id: cart-session-a3f1
session_ttl: 1800

session_id is the handle. Requests that share it reuse the same proxy IP. session_ttl is 1800 seconds unless you raise it (max 86400). proxy_sticky keeps the exit IP for the browser session on that job. After the session ends, the next id draws a different member of the pool.

Reuse one id for cookie-jar follow-ups, then mint a new id for the next independent shopper. Start from the catalog scrape_body and overlay the sticky keys:

import os
import uuid
import requests

API = os.environ["SCRAPE_API"]
headers = {
    "Authorization": f"Bearer {os.environ['SCRAPE_API_KEY']}",
    "Content-Type": "application/json",
}
profile_id = int(os.environ["PROXY_PROFILE_ID"])

def scrape(url, session_id):
    r = requests.post(
        f"{API}/api/v1/scrape",
        headers=headers,
        json={
            "url": url,
            "method": "GET",
            "formats": ["text"],
            "proxy_profile_id": profile_id,
            "proxy_sticky": True,
            "session_id": session_id,
            "session_ttl": 1800,
        },
    )
    r.raise_for_status()
    return r.json()

session_a = f"cart-{uuid.uuid4().hex[:12]}"
scrape("https://store.example.com/cart", session_a)
scrape("https://store.example.com/cart", session_a)  # same exit IP, cookie jar stays valid

session_b = f"cart-{uuid.uuid4().hex[:12]}"
scrape("https://store.example.com/cart", session_b)  # new session, different pool member

Sticky sessions waste pool capacity if you pin them on a 10k-URL catalog crawl. Use them where the target actually fingerprints IP with the cookie. For longer multi-step flows, keep one exit IP across multi-step scrape requests rather than turning the entire profile sticky. Broader rotation patterns belong in proxy rotation strategies, not in a cart session.

Skip a dead proxy in your pool without abandoning the rest of the job

Owned pools fail in ordinary ways: one datacenter box returns 502, a residential gateway times out, a stale password yields a proxy 407 on your endpoint. Fail-fast kills the job. Failover skips the sick member and continues. auto_skip_failed: true on the profile is the API-side switch. max_retries on the scrape body is the per-request budget. Neither field is a backoff timer and neither quarantines an IP for N seconds — implement that in the worker if you need it.

Operational policy next to the profile (what the API accepts, plus what you run locally):

name: store-allowlist-dc
rotation_mode: round_robin
auto_skip_failed: true   # accepted on POST/PUT /api/v1/proxy-profiles
scrape_defaults:
  max_retries: 5         # accepted on POST /api/v1/scrape
worker_policy:           # your runner, not an API field
  backoff_seconds: [1, 2, 4]
  remove_unhealthy_for_seconds: 120

Do not retry on HTTP 407 from POST /api/v1/scrape. That status, if you ever see it, is the scrape API’s own response, not your fleet proxy talking. Treat proxy 407/502/timeout as something you detect when testing members of the pool, or as a failed scrape result you resubmit so auto_skip_failed can pick the next endpoint.

Local health check: walk the DC pool against the echo endpoint, not the catalog. A store 403 must not quarantine a healthy box.

import ipaddress
import os
import time
import requests

password = os.environ["PROXY_PASS_DC"]
candidates = [
    f"http://dc_user_a:{password}@dc1.proxy.example.com:8080",
    f"http://dc_user_b:{password}@dc2.proxy.example.com:8080",
]
unhealthy_until = {}

def probe(proxy_url):
    try:
        r = requests.get(
            "https://api.example.com/ip",
            proxies={"http": proxy_url, "https": proxy_url},
            timeout=10,
        )
        if r.status_code in (407, 502) or r.status_code >= 500:
            return False
        r.raise_for_status()
        ipaddress.ip_address(r.json()["ip"])
        return True
    except (requests.RequestException, ValueError, KeyError):
        return False

now = time.time()
live = []
for url in candidates:
    until = unhealthy_until.get(url, 0)
    if now < until:
        continue
    if probe(url):
        live.append(url)
    else:
        unhealthy_until[url] = now + 120

if not live:
    raise SystemExit("no healthy endpoints in store-allowlist-dc")

Then submit the catalog job against the profile so a member that dies during the crawl is skipped instead of ending the batch. Same scrape_body as the attach section; add retry budget only:

scrape_body["max_retries"] = 5
scrape_body["auto_retry"] = True
scrape_body["timeout"] = 60
StrategyOne endpoint deadJob outcome
Fail-fastFirst member 502/timeoutJob dies, catalog unfinished
Pool failover (auto_skip_failed + retries)First member 502/timeoutNext member used, job completes with a smaller pool

Turning auto_skip_failed off is reasonable when the pool has two IPs and you would rather page than silently run on one. For anything larger, fail-fast is how a single bad box wastes a crawl window. Quarantine locally for 120 seconds, keep max_retries at 5, and do not confuse a target 403 (allowlist miss) with a proxy 502 (your box is down). Those two errors demand opposite fixes: 403 means the exit was not in 203.0.113.0/24 (wrong profile, or a residential hop you swore you did not mix in); 502 means skip that member and stay on store-allowlist-dc.

Wire the DC profile to the job, keep API keys and proxy passwords in the environment, and measure egress on an echo URL before you raise concurrency. Allowlisted targets will accept the catalog request when the exit IP is actually yours; they will keep returning 403 when it is not. Split allowlist datacenter IPs from rotating residential, pin session_id only where cookies follow the IP, and let a dead member drop out of the pool instead of taking the job with it.

#bring-your-own-proxy #proxy-profile #allowlist #web-scraping #residential-proxies #slot:api-capability

Related Articles

Tutorial

Scrape Localized Storefronts With a Country Exit Code

Storefronts change price, language, and stock by visitor country. Pin the scrape exit with an ISO-2 code and handle 422 when the country is unsupported.

 Tutorial

Keep One Exit IP Across Multi-Step Scrape Requests

Learn how proxy_sticky keeps the same exit IP for warmup and follow-up requests, preventing session resets during multi-step scraping workflows.

 Tutorial

Sticky Exit IPs for Warmup and Follow-Up Scrapes

Rotating proxies between a warmup hit and the next request breaks cookies and geo checks. Learn how a sticky exit IP keeps the session on the same IP.