Scrape Localized Storefronts With a Country Exit Code

Why store.example.com Changes Price, Language, and Stock by Visitor Country

You request https://store.example.com/products/sku-1001 from a CI runner sitting on a generic datacenter IP. The payload comes back at $49.00, English copy, in stock. A teammate in Berlin opens the same path and sees 44,90 €, German copy, and a backorder badge. Same SKU. Same URL. Different catalog.

Storefronts do not treat a product URL as a global document. They pick a market from the exit IP, then layer cookies, Accept-Language, and currency prefs on top. Price, tax, language, and the in-stock flag are functions of that market, not of the path. Unpinned scrapes inherit whatever country the proxy pool happens to land in. You do not get “the” product. You get one region’s view of it, unlabeled.

Naive fixes fail in a predictable way. Setting Accept-Language: de-DE without pinning the exit still leaves GeoIP in charge, so the store keeps serving the IP’s market and maybe translates a few strings. Forcing ?currency=EUR on a US exit often produces a converted number next to English copy and US stock, which is worse than a wrong market: it looks consistent and is not. If you are building price intelligence, that silent mix is how bad rows enter a warehouse.

The same SKU diverges across three real markets like this:

Exit market	Currency	Language	In stock
US	USD	English (en-US)	yes
DE	EUR	German (de-DE)	yes
JP	JPY	Japanese (ja-JP)	no

A default fetch with no country pin produces the mismatch in code. The request succeeds. The currency is still wrong.

import requests

SKU_URL = "https://store.example.com/products/sku-1001"

resp = requests.get(SKU_URL, timeout=60)
resp.raise_for_status()
print(resp.status_code)
print(resp.headers.get("Content-Language"))
print(resp.text[:400])

# Illustrative unpinned body (CI runner, no country pin):
# html[lang]=ja-JP, .price-currency=JPY, .product-price=7280,
# .stock-flag=out_of_stock
# HTTP 200 means the fetch ran. It does not mean the payload belongs
# to the market you intend to store.

Pin the Scrape Exit Node With an ISO-2 Country Code

Force the TCP exit into the market you care about. The storefront’s GeoIP lookup then sees a German address, a US address, or a Japanese address, and selects VAT, copy, and warehouse rules for that country. ISO-3166 alpha-2 is the right grain: DE, not Germany, not EU, not a city name.

Do that with a country-coded proxy, Accept-Language, and a locale query on the same request. Most residential providers encode the ISO-2 in the username (customer-country-DE) or in a subdomain (de.gw.…). Pair the pin with a residential exit when the storefront’s GeoIP tables are tuned for consumer ISPs. Datacenter ranges get classified as “unknown” or “VPN” on a lot of shop stacks, and “unknown” often falls through to a default catalog (commonly US or the shop’s HQ country). Residential costs more. I still prefer it here, because a cheap datacenter pin that the shop ignores is a fake pin.

curl -sS \
  --proxy "http://customer-country-DE:secret@gw.isp-proxy.net:8000" \
  -H "Accept-Language: de-DE,de;q=0.9" \
  -H "Accept: text/html,application/json;q=0.9" \
  -H "X-Requested-Currency: EUR" \
  "https://store.example.com/products/sku-1001?locale=de-DE"

Keep the allowed set explicit. Do not interpolate a user-supplied country string straight into the proxy username. A typo (UK instead of GB, GER instead of DE) is how you burn retries on codes the fleet will never honor.

One MARKET table is the allowlist and the locale/currency/language triple. Proxy URL, query string, headers, fallbacks, and validation all read this object so those maps cannot drift apart.

from typing import Optional

import requests

SKU_URL = "https://store.example.com/products/sku-1001"
PROXY_TMPL = "http://customer-country-{country}:secret@gw.isp-proxy.net:8000"

# ISO-2 codes this scraper is allowed to pin. The key is the exit.
# locale / currency / accept_language travel with that exit.
MARKET = {
    "US": {
        "locale": "en-US",
        "currency": "USD",
        "accept_language": "en-US,en;q=0.9",
    },
    "DE": {
        "locale": "de-DE",
        "currency": "EUR",
        "accept_language": "de-DE,de;q=0.9",
    },
    "FR": {
        "locale": "fr-FR",
        "currency": "EUR",
        "accept_language": "fr-FR,fr;q=0.9",
    },
    "JP": {
        "locale": "ja-JP",
        "currency": "JPY",
        "accept_language": "ja-JP,ja;q=0.9",
    },
    "GB": {
        "locale": "en-GB",
        "currency": "GBP",
        "accept_language": "en-GB,en;q=0.9",
    },
    "IE": {
        "locale": "en-IE",
        "currency": "EUR",
        "accept_language": "en-IE,en;q=0.9",
    },
    "AT": {
        "locale": "de-AT",
        "currency": "EUR",
        "accept_language": "de-AT,de;q=0.9,en;q=0.8",
    },
}

def exit_for(iso2: str) -> str:
    code = iso2.upper()
    if code not in MARKET:
        raise ValueError(f"unsupported scrape market: {iso2}")
    return code

def proxy_url(iso2: str) -> str:
    return PROXY_TMPL.format(country=exit_for(iso2))

def product_url(iso2: str) -> str:
    return f"{SKU_URL}?locale={MARKET[exit_for(iso2)]['locale']}"

def market_headers(iso2: str) -> dict[str, str]:
    market = MARKET[exit_for(iso2)]
    return {
        "Accept-Language": market["accept_language"],
        "Accept": "text/html,application/json;q=0.9",
        "X-Requested-Currency": market["currency"],
    }

def fetch_storefront(iso2: Optional[str] = None) -> requests.Response:
    """Single request shape for sku-1001: country proxy, locale, headers.

    iso2=None is the unpinned control (no proxy, no locale, no language).
    422 is a destination signal — do not raise it here.
    """
    if iso2 is None:
        resp = requests.get(
            SKU_URL,
            headers={"Accept": "text/html,application/json;q=0.9"},
            timeout=60,
        )
    else:
        country = exit_for(iso2)
        proxy = proxy_url(country)
        resp = requests.get(
            product_url(country),
            headers=market_headers(country),
            proxies={"http": proxy, "https": proxy},
            timeout=60,
        )
    if resp.status_code >= 500 or resp.status_code in (408, 429):
        resp.raise_for_status()
    return resp

Confirm the provider actually egressed from the requested country (dashboard or an IP echo) on the first run. After that, catalog fields are the proof: if DE still returns USD, the pin did not take.

A scrape API is one implementation of the same pin, not a different lesson. FineData maps the ISO-2 to proxy_country and the ISP-grade exit to use_residential. If you already have a country-coded proxy, skip this block; every later snippet uses fetch_storefront.

# Optional mapping only. Same MARKET, same headers, same locale query.
FINEDATA = "https://api.finedata.ai/api/v1/scrape"

def scrape_body_via_api(iso2: str) -> dict:
    country = exit_for(iso2)
    return {
        "url": product_url(country),
        "proxy_country": country,
        "use_residential": True,
        "headers": market_headers(country),
    }

Pinning the exit country is not the same as holding one IP across a login or cart flow. Country selects the market. A sticky session selects a single address inside that market. If you reuse a sticky session minted on a US fetch while sending a DE proxy, you are asking for two contradictory routing rules. Mint a new sticky session per market, or skip stickiness for one-shot product fetches.

Treat HTTP 422 as an Unsupported-Country Signal, Not a Hard Failure

Some storefronts refuse to render a market they do not sell into. A common pattern is HTTP 422 with a JSON body that names the country it rejected. That is not a transport failure, not a selector bug, and not a reason to mark the scraper “down”. It is a geo-availability miss.

Treat the JSON below as a hypothetical contract, not as a documented fact about store.example.com. Probe the live shop before you branch on it. Other shops 404, 302 to a home-market URL, or 200 an empty grid for the same miss.

{
  "error": "unsupported_country",
  "unsupported_country": "XX",
  "message": "Catalog is not offered in the requested market"
}

Because this client talks to the shop through a proxy, the HTTP status on resp is the shop’s status. Branch on that, plus an explicit error key if the body is JSON. Do not treat a generic status field inside a JSON object as proof of a 422: scrape-API envelopes reuse that key for job state, and colliding on it will page you for successful fetches.

If you do go through a scrape API, a 200 from the API means the job ran. The destination 422, when it happens, shows up in the wrapped content. Unwrap that body. Do not raise_for_status() yourself into a pager because Austria is not a listed shipping country.

import json
from typing import Any, Optional

from bs4 import BeautifulSoup

class UnsupportedCountry(Exception):
    def __init__(self, iso2: str, payload: dict):
        super().__init__(f"storefront rejected market {iso2}")
        self.iso2 = iso2
        self.payload = payload

def _text(soup: BeautifulSoup, selector: str) -> Optional[str]:
    node = soup.select_one(selector)
    return node.get_text(strip=True) if node else None

def extract_fields(document: str) -> dict:
    soup = BeautifulSoup(document, "html.parser")
    html_tag = soup.find("html")
    lang = html_tag.get("lang") if html_tag and html_tag.has_attr("lang") else None
    return {
        "title": _text(soup, "h1"),
        "price": _text(soup, ".product-price"),
        "currency": _text(soup, ".price-currency"),
        "availability": _text(soup, ".stock-flag"),
        "lang": lang,
    }

def parse_response(resp: requests.Response) -> dict:
    """Shop body only. HTML product page or JSON error object."""
    raw = resp.text or ""
    ctype = (resp.headers.get("Content-Type") or "").lower()
    looks_json = "json" in ctype or raw.lstrip().startswith("{")
    if looks_json and raw.strip():
        try:
            data = json.loads(raw)
        except json.JSONDecodeError:
            data = None
        if isinstance(data, dict):
            return data
        if "json" in ctype:
            raise ValueError("storefront JSON was not an object")
    if not raw:
        raise ValueError("empty storefront body")
    return extract_fields(raw)

def rejected_market(resp: requests.Response, payload: dict, iso2: str) -> Optional[str]:
    """ISO-2 the shop refused, or None.

    HTTP 422 is the primary signal. `error: unsupported_country` is the
    hypothetical JSON contract above — confirm it on the live host.
    Never key off a bare `status` field; too many envelopes own that name.
    """
    body_miss = payload.get("error") == "unsupported_country"
    if resp.status_code != 422 and not body_miss:
        return None
    return payload.get("unsupported_country") or payload.get("country") or iso2

def scrape_sku(iso2: str) -> dict:
    country = exit_for(iso2)
    resp = fetch_storefront(country)
    payload = parse_response(resp)
    rejected = rejected_market(resp, payload, country)
    if rejected:
        print(f"rejected ISO-2={rejected} http={resp.status_code} body={payload}")
        raise UnsupportedCountry(rejected, payload)
    payload["resolved_country"] = country
    return payload

Retrying the same ISO-2 after a 422 is wasted work. The shop already told you that market does not exist. Back off, record the miss, and either skip the row or walk a fallback chain (next section). Treat 5xx, 429, and empty bodies as retryable. Treat 422 as terminal for that country.

Align Accept-Language, Locale, and Currency With the Pinned ISO-2 Code

Exit country alone is not enough. Plenty of shops GeoIP you into Germany, then still render English because Accept-Language says en-US. Others lock currency to a cookie or a locale query param and ignore the IP after the first hop. If those signals disagree, you get German VAT on English copy, or EUR prices with a $ glyph. Downstream, that looks like a successful scrape.

Build one mapping and apply it as a unit. That mapping is MARKET above. fetch_storefront already attaches locale, Accept-Language, and X-Requested-Currency from the same row. There is no second request shape for this SKU: do not stand up a JSON-only client (Accept: application/json, no locale) beside an HTML client. HTML vs JSON is a response problem (parse_response), not a request one.

def scrape_pinned(iso2: str) -> dict:
    # Same SKU, same fetch_storefront() shape as scrape_sku.
    return scrape_sku(iso2)

for iso2 in ("US", "DE", "JP"):
    print(
        iso2,
        product_url(iso2),
        market_headers(iso2)["Accept-Language"],
        MARKET[iso2]["currency"],
    )

Do not copy Accept-Language from your laptop. Local browsers send en-US,en;q=0.9 even when you think you are testing DE. Pinning a DE proxy and then leaking a US language header is how you “confirm” the pin while still storing English titles. Drive both values from MARKET.

One trade-off: some shops ignore Accept-Language entirely and key only on IP plus a locale cookie set by a previous page. Those need a two-step scrape (home, then product) with a sticky session so the cookie and the exit IP stay together. That is slower. For store.example.com, the header-plus-query combination is enough, and I would not add a warmup hop until a measurement (below) shows language still drifting.

Retry With a Neighbor Market When the Exit Country Returns 422

A 422 should not always kill the job. Coverage pipelines (does this SKU exist anywhere in DACH?) can walk a neighbor chain and still return a row. Pricing pipelines should be stricter. Mixing markets is how you invent a discount that never existed.

If you do fall back, make the chain deterministic and tiny. Record the ISO-2 that actually succeeded, not the one you were asked for. Stamping AT on a payload that came from DE is a lie your analysts will treat as truth.

Requested	Fallback 1	Fallback 2
AT	DE	US
GB	IE	US
IE	GB	US
FR	DE	US
JP	US	—

US sits at the end because store.example.com always publishes a US catalog. That is a shop-specific fact, not a universal rule. Do not cargo-cult “fall back to US” onto a merchant that 422s every non-home market.

FALLBACK_CHAINS = {
    "AT": ["AT", "DE", "US"],
    "GB": ["GB", "IE", "US"],
    "IE": ["IE", "GB", "US"],
    "FR": ["FR", "DE", "US"],
    "JP": ["JP", "US"],
    "DE": ["DE", "US"],
    "US": ["US"],
}

def scrape_with_fallback(requested: str) -> dict:
    chain = FALLBACK_CHAINS.get(requested.upper(), [requested.upper(), "US"])
    last_exc: Optional[UnsupportedCountry] = None

    for iso2 in chain:
        try:
            payload = scrape_sku(iso2)
        except UnsupportedCountry as exc:
            print(f"chain miss requested={requested} tried={iso2}")
            last_exc = exc
            continue

        payload["requested_country"] = requested.upper()
        payload["resolved_country"] = iso2
        print(f"chain hit requested={requested} resolved={iso2}")
        return payload

    raise last_exc or RuntimeError(f"no market in chain for {requested}")

I prefer a missing AT row over a DE price stamped as AT. Pricing models handle silence. They do not handle a 3% “Austria” gap that is actually German list price with different VAT. If a product manager wants coverage more than purity, keep the fallback, but make resolved_country a required column and never join on requested_country for money fields. Teams that skip that column will disagree with this. They are the ones who later debug phantom discounts.

Cap the chain at three hops. Each hop is a full residential fetch. Four neighbors is not resilience; it is a latency budget on fire.

Measure Price, Language, and Stock Drift Between Unpinned and Country-Pinned Scrapes

Do not trust the pin because the request returned 200. Prove it changed the payload. Scrape the same SKU three ways: no country proxy, US, and FR. Compare price, currency, language, and stock. If unpinned equals US, your default pool is already American and every “global” scrape you ran last month was a US scrape. If unpinned equals none of the pinned runs, the pool is wandering and your historical table is a blend.

RUNS = [
    ("unpinned", None),
    ("US", "US"),
    ("FR", "FR"),
]

def measure(sku_url: str) -> list[dict]:
    global SKU_URL
    sku_url = sku_url or SKU_URL
    previous, SKU_URL = SKU_URL, sku_url
    try:
        rows = []
        for label, iso2 in RUNS:
            resp = fetch_storefront(iso2)
            extracted = parse_response(resp)
            rows.append(
                {
                    "run": label,
                    "proxy_country": iso2,
                    "price": extracted.get("price"),
                    "currency": extracted.get("currency"),
                    "language": extracted.get("lang"),
                    "availability": extracted.get("availability"),
                }
            )
        return rows
    finally:
        SKU_URL = previous

for row in measure("https://store.example.com/products/sku-1001"):
    print(row)

Local run against sku-1001 (illustrative payload, same extractor, three exits):

Run	proxy_country	Price	Currency	Language	In stock
unpinned	—	7280	JPY	ja-JP	no
country=US	US	49.00	USD	en-US	yes
country=FR	FR	45,00	EUR	fr-FR	yes

Highlighted fields are the ones that moved. Price magnitude, currency code, html[lang], and the stock flag all flipped with the exit. That is the evidence the pin is doing work. If FR and US had matched, the shop would be ignoring GeoIP and you would be paying for a residential country route that does nothing — stop pinning and look at locale cookies instead.

Re-run this measurement when the merchant ships a new storefront, not on a calendar. A redesign that starts keying off CF-IPCountry-style headers, or that collapses EU into one EUR catalog, will invalidate the table above without changing your code.

Assert Currency and Locale Match the Exit Code Before Writing Catalog Rows

HTTP 200 with a pin still is not a write. The shop can 200 a US page through a DE exit when a CDN cache key omits country, or when a leftover sticky session carries a US cookie onto a German IP. Guard the insert. If currency or language disagrees with the market object, drop the row and emit a metric. Silent mismatch is how a warehouse rots.

Read expectations from MARKET. Do not keep a second table of expected currencies or a third “validation config” that copies the same rows — those copies will drift.

class GeoMismatch(Exception):
    pass

def assert_market(iso2: str, extracted: dict) -> None:
    spec = MARKET[exit_for(iso2)]
    currency = (extracted.get("currency") or "").upper()
    lang = extracted.get("lang") or extracted.get("language") or ""

    if currency != spec["currency"]:
        raise GeoMismatch(
            f"{iso2}: currency {currency!r} != {spec['currency']!r}"
        )
    if not lang.lower().startswith(spec["locale"].lower()[:2]):
        # Accept de-DE or de; reject en-US on a DE pin.
        raise GeoMismatch(
            f"{iso2}: language {lang!r} does not match {spec['locale']!r}"
        )

def ingest(iso2: str, extracted: dict, db) -> None:
    resolved = extracted.get("resolved_country") or iso2
    assert_market(resolved, extracted)
    db.insert(
        {
            "sku": "sku-1001",
            "requested_country": iso2,
            "resolved_country": resolved,
            "price": extracted["price"],
            "currency": extracted["currency"],
            "language": extracted.get("lang"),
            "in_stock": extracted.get("availability"),
        }
    )

Policy sits next to the functions, not as a duplicate market map: on mismatch, reject the row; on 422, mark the market unavailable. Prefix-match language (de for de-DE / de-AT) if the shop only sets html[lang]=de. Do not prefix-match currency. EU is not a currency. € as a glyph with no ISO code is a parser bug, not a market. Fail those closed.

Wire mismatch to reject, not to “fix” the currency in software. Converting 49 USD to EUR at yesterday’s rate is not a DE catalog row. It is FX, and it will not match the tax-inclusive number a Berlin customer sees.

Pin, Align, Then Refuse the Wrong Catalog

Unpinned fetches of store.example.com/products/sku-1001 are market lottery tickets. Pin the TCP exit to an ISO-2 code (country-coded proxy; FineData proxy_country if that is your transport), send Accept-Language and locale from the same MARKET map, and treat destination 422 as “this country is not sold,” not as scraper failure. Walk a short neighbor chain only if you persist resolved_country. Assert currency and language before insert so a 200 cannot smuggle a JP payload into a US partition.

Measure unpinned vs pinned on one SKU before you roll this out to the rest of the catalog. If the three-way table does not diverge, the pin is not the lever — stop spending residential traffic on it and go chase locale cookies instead.