Steam 429 Error: Parsing Store Pages Responsibly with Python

A 429 Too Many Requests from store.steampowered.com means exactly what RFC 6585 §4 says: the origin decided you sent too many requests in a given window and it MAY tell you how long to wait via Retry-After. Steam does not publish a single numeric threshold for how many Store-page requests trigger that response, and this article does not invent one. The API Terms of Use limit you to 100,000 Steam Web API calls per day. That is a limit on use of the official Web API, not a published Store-page request budget or an allowance to multiply by creating more keys. It says nothing about the rate at which the public Store HTML (/app/<id>/) or the unauthenticated api/appdetails endpoint will return 429 — those are a different surface with undocumented, empirically-observed-only limits, and this article does not publish empirical numbers either.

This piece covers three things: what a 429 actually looks like in a scrape response envelope, how to build a Python client that defers correctly instead of retrying before the server’s wait expires or rotating IPs to dodge the block, and what Steam’s own terms say about automation before you point any of this at their pages.

Three Different “Steam APIs,” One Article

Before writing a single request, separate what you are actually hitting:

• Steam Store HTML (store.steampowered.com/app/<appid>/...) — public, unauthenticated, rendered for browsers. This is what the example below scrapes.
• Official Steam Web API (api.steampowered.com/...) — key-authenticated, documented, rate-limited at 100,000 calls/day. Use this instead of scraping HTML whenever the data you need (owned games, player counts, app news) is exposed there — the Web API overview has the endpoint list.
• Steam Community Market — a third surface with its own separate throttling behavior, not covered here.

Conflating these three is how “Steam rate limits” articles end up publishing a number that only ever applied to one of them. If the official API covers your use case, call it directly with your key; do not scrape the page that key was meant to replace.

What a 429 Looks Like in the Response

A scrape wraps every request in an outer HTTP response plus a body. The outer HTTP status is not the target site’s status — success and status_code inside the body are what tells you what actually happened at Steam:

{
  "success": false,
  "status_code": 429,
  "headers": {"retry-after": "3600"},
  "body": "...",
  "tokens_used": 0
}

success: false with status_code: 429 means the target rejected the request. A 200 with success: false (covered in the Wikipedia scraping companion article) means something else entirely: the page loaded but tripped a block heuristic. Blind-retrying on any non-success response without reading status_code first is how a client turns one rate-limit into a retry storm against the same 429.

There is a third failure mode neither of those covers: the outer HTTP call itself can fail (timeout, 5xx, malformed response) before a target status_code ever exists. That is not the same as a target 429 and must not be handled by the same retry path — treating an outer transport error as “the target rate-limited me” throws away a real infrastructure signal.

Honor Retry-After without Blocking a Worker

RFC 9110 §10.2.3 defines two legal forms for the header: delay-seconds (a bare non-negative integer) or an HTTP-date in IMF-fixdate form (Sat, 31 Oct 2026 18:00:00 GMT). A conforming client parses either form and waits at least that long — it does not clamp the delay down to something more convenient and retry early. wait_s = min(retry_after, 120) followed by an immediate second request is a real bug, not a simplification: if Steam says Retry-After: 3600 and the client waits 120 seconds anyway, it has just sent request number two into the same rate limit window, one violation closer to a harder block.

The correct shape has two branches. If the delay is short enough to hold a request thread open for (below some max_inline_wait, e.g. 30 seconds), one bounded retry after sleeping the full delay is reasonable. If the delay is longer than that, the client should not sleep at all — it should surface a deferred-retry signal (a retry_at timestamp or delay seconds) so the caller reschedules the job, instead of blocking a worker or thread for an hour. Switching proxy IP to avoid either path is not a Retry-After strategy; it is the thing rate limiting exists to stop, and on Steam specifically it also runs into the contractual point below.

These examples use auto_retry: false and max_retries: 1. A live request with max_retries: 0 was rejected with HTTP 422 because the parameter’s minimum is 1. Application-level retry handling below is explicit: one short wait and retry, or a deferred exception for the caller. Exception handling is not a queue implementation; a worker receiving DeferredRetry must persist a not-before time and enforce its own total retry budget. An impractically large header stops processing for review rather than being shortened.

A Reusable Python Client

Install httpx, export FINEDATA_API_KEY, and save the following example as steam_store.py. This client posts to FineData’s scrape endpoint, separates outer HTTP failures from target-site failures, parses Retry-After per RFC 9110 (case-insensitive header lookup, UTC-aware HTTP-dates, a 30-second fallback for missing or invalid values), and either retries once inline or raises a DeferredRetry for the caller to reschedule — it never retries on its own past max_inline_wait. It also demonstrates an appid/locale/country cache key, since Store pages vary by both l= and cc= query params and a cache keyed only on appid will silently serve the wrong locale’s price.

import os
import time
from email.utils import parsedate_to_datetime
from datetime import datetime, timezone

import httpx

API_KEY = os.environ["FINEDATA_API_KEY"]
BASE_URL = os.environ.get("FINEDATA_API_BASE", "https://api.finedata.ai")


class DeferredRetry(Exception):
    """Raised instead of sleeping when Retry-After exceeds max_inline_wait.
    The caller decides how to reschedule (queue, cron, delayed task)."""

    def __init__(self, retry_at: float, delay_s: float):
        self.retry_at = retry_at  # unix timestamp
        self.delay_s = delay_s
        super().__init__(f"Deferred: retry at {retry_at} (delay {delay_s}s)")


def _parse_retry_after(headers: dict, now: float | None = None) -> float:
    """RFC 9110 10.2.3: delay-seconds (digits) or an HTTP-date. Header lookup
    is case-insensitive. Missing, unparseable, or malformed -> a conservative
    30s fallback, never an immediate retry."""
    now = time.time() if now is None else now
    value = None
    for k, v in headers.items():
        if k.lower() == "retry-after":
            value = v
            break
    if not value:
        return 30.0
    value = str(value).strip()
    if value.isascii() and value.isdigit():
        seconds = float(value)
        # Do not shorten an impractical wait: stop for operator review.
        if seconds > 10**8:
            raise RuntimeError("Retry-After too large to schedule; stop for review")
        return seconds
    try:
        dt = parsedate_to_datetime(value)
        if dt.tzinfo is None:
            dt = dt.replace(tzinfo=timezone.utc)
        delta = (dt - datetime.fromtimestamp(now, tz=timezone.utc)).total_seconds()
        return max(0.0, delta)
    except (TypeError, ValueError, OverflowError):
        return 30.0


def cache_key(appid: int, locale: str, country: str) -> str:
    """Store pricing/localization vary by both params - key on all three."""
    return f"{appid}/{locale}/{country}"


def _post_scrape(client: httpx.Client, base_url: str, api_key: str, payload: dict) -> dict:
    """One outer HTTP call. Raises on transport failure or a non-JSON body -
    both are infrastructure problems, distinct from a target-site rejection."""
    resp = client.post(
        f"{base_url}/api/v1/scrape",
        headers={"Authorization": f"Bearer {api_key}"},
        json=payload,
    )
    try:
        data = resp.json()
    except ValueError as exc:
        raise RuntimeError(
            f"Non-JSON response body, outer HTTP {resp.status_code}"
        ) from exc

    if not 200 <= resp.status_code < 300:
        # An outer failure stays separate even if its JSON resembles a scrape.
        raise RuntimeError(f"Transport error: outer HTTP {resp.status_code}")

    if not isinstance(data, dict) or type(data.get("success")) is not bool:
        raise RuntimeError("Malformed scrape response: missing 'success' boolean")
    status = data.get("status_code")
    if type(status) is not int:
        raise RuntimeError("Malformed scrape response: invalid target status")
    if data["success"] and not 200 <= status < 300:
        raise RuntimeError("Inconsistent scrape success and target status")

    return data


def fetch_app_page(
    appid: int,
    locale: str = "english",
    country: str = "us",
    extract_rules: dict | None = None,
    max_inline_wait: float = 30.0,
    api_key: str = API_KEY,
    base_url: str = BASE_URL,
    client: httpx.Client | None = None,
    sleep=time.sleep,
) -> dict:
    """Bounded single retry on a short 429, DeferredRetry on a long one.
    Raises RuntimeError on a repeated 429 or a non-200 success:false with a
    block label - never returns a failed result as if it were data."""
    if client is None:
        with httpx.Client(timeout=60) as managed_client:
            return fetch_app_page(
                appid, locale, country, extract_rules, max_inline_wait,
                api_key, base_url, managed_client, sleep,
            )
    own_client = client
    url = f"https://store.steampowered.com/app/{appid}/?l={locale}&cc={country}"
    payload = {
        "url": url,
        "formats": ["markdown"],
        "only_main_content": True,
        "max_retries": 1,
        "auto_retry": False,
        "use_antibot": False,
        "timeout": 45,
    }
    if extract_rules:
        payload["extract_rules"] = extract_rules

    data = _post_scrape(own_client, base_url, api_key, payload)

    if data.get("status_code") == 429 and not data.get("success"):
        wait_s = _parse_retry_after(data.get("headers", {}) or {})
        if wait_s > max_inline_wait:
            raise DeferredRetry(retry_at=time.time() + wait_s, delay_s=wait_s)
        sleep(wait_s)
        data = _post_scrape(own_client, base_url, api_key, payload)
        if data.get("status_code") == 429 and not data.get("success"):
            raise RuntimeError("Repeated 429 after one bounded retry - back off, do not loop")

    if not data.get("success"):
        label = (data.get("meta") or {}).get("block_reason") or data.get("status_code")
        raise RuntimeError(f"Scrape did not succeed (status={data.get('status_code')}, block={label})")

    output = data.get("data")
    extract = output.get("extract") if isinstance(output, dict) else None
    if extract_rules and (
        not isinstance(extract, dict)
        or any(not isinstance(extract.get(key), str) or not extract[key].strip()
               for key in ("title", "purchase_blocks"))
    ):
        raise RuntimeError("Required fields 'title'/'purchase_blocks' missing or empty in extract")

    return data


if __name__ == "__main__":
    result = fetch_app_page(
        appid=620,
        locale="english",
        country="us",
        extract_rules={
            "title": "#appHubAppName",
            "purchase_blocks": ".game_area_purchase_game",
        },
    )
    print(result.get("status_code"), result.get("data", {}).get("extract"))

Live Evidence, Measured 2026-09-05

The live checks used the MCP interface, not this Python process. Its retry and validation branches are tested with synthetic responses. Against https://store.steampowered.com/app/620/Portal_2/?l=english&cc=us, a plain scrape with formats: ["markdown"], only_main_content: true, max_retries: 1, auto_retry: false, use_antibot: false returned success: true, status_code: 200, 1 token, ~2.2s. Adding the extract_rules shown above returned, verbatim:

{"title": "Portal 2", "purchase_blocks": "Buy Portal 2$9.99Add to Cart"}

That purchase_blocks string is the concatenated text of the first element matching .game_area_purchase_game — Portal 2’s store page repeats this block per purchase option (base game, bundles), and a plain string selector in extract_rules returns only the first match, not a list. If a page has multiple purchase blocks and you need all of them, use the advanced form: {"selector": ".game_area_purchase_game", "type": "list"}. No live 429 was observed or deliberately provoked during this check; the 429/Retry-After request-response bodies shown earlier are minimal synthetic illustrations built from the documented response shape, not a reproduced rate-limit.

What Steam’s Own Terms Say About Automation

Read §4.C of the Steam Subscriber Agreement directly before automating any interaction with Steam — do not rely on a paraphrase:

“You may not use any form of scripts, bots, macros, or other non-human-controlled systems (‘Automation’) to interact with Content and Services on Steam in any manner, including but not limited to…”

The opening clause — “in any manner” — is not scoped to accounts or gameplay; the illustrative list after it is examples, not the boundary of the prohibition. Public availability of a page, or requesting it at a low, human-comparable rate, does not by itself grant permission under this clause — whether any automated access is permitted depends on rights you separately have (e.g. a license, a partnership, or the official Web API’s own terms), not on how quietly you scrape.

Sources and Evidence Boundary

• Steam Subscriber Agreement §4 — Online Conduct, Cheating and Automation; §4.C quoted above verbatim.
• Steam Web API Terms of Use — states the 100,000 calls/day limit for the official, key-authenticated Web API.
• Steamworks Web API Overview — official endpoint documentation.
• RFC 6585 §4 — defines the 429 Too Many Requests status code.
• RFC 9110 §10.2.3 — defines the Retry-After header’s delay-seconds and HTTP-date forms.

All source claims above were verified read-only against the linked pages. The Portal 2 extraction result (title, purchase_blocks) is a live measurement taken 2026-09-05; the 429/Retry-After bodies shown earlier are synthetic illustrations of the documented schema, not a reproduced rate-limit event. For related patterns, see structured extraction with extract_rules, extract_schema, and extract_prompt and async jobs for larger catalog runs.