Hidden JSON Endpoints vs DOM Parsing: What to Scrape

Where the Data Actually Lives: Hydration Scripts, State Tags, and XHR Endpoints on a Modern Product Page

Open any modern storefront product page and view the source. Before you write a single CSS selector, count how many times structured data appears in the raw HTML. On a typical React, Next.js, or Vue storefront, the answer is: at least twice, before the browser fires a single XHR call.

Here is what a product page from store.example.com looks like under the hood, stripped down to the parts that matter:

<div class="price-card">
  <span class="price-card__final">$1,299.00</span>
  <span class="availability-label">In Stock</span>
</div>

<script id="__NEXT_DATA__" type="application/json">
{
  "props": {
    "pageProps": {
      "product": {
        "id": 123,
        "title": "Ultra Widget Pro",
        "pricing": { "cents": 129900, "currency": "USD" },
        "inventory": { "qty": 14, "warehouse": "us-east" },
        "variants": [
          { "sku": "UWP-BLK-64", "cents": 129900 },
          { "sku": "UWP-BLK-256", "cents": 154900 }
        ]
      }
    }
  }
}
</script>

<script>window.__APP_STATE__ = {"cart": {"items": 0}, "session": {"country": "US"}};</script>

Three data sources, one page:

1. The rendered DOM — what a headless browser sees after hydration. $1,299.00 as a string, “In Stock” as a label.
2. Hydration/state scripts — __NEXT_DATA__, __APP_STATE__, __NUXT__, window.__INITIAL_STATE__. Full JSON objects embedded in <script> tags, present in the very first HTTP response. No JS rendering required.
3. XHR/fetch endpoints — the API the frontend itself calls. Paginated grids, filters, and search almost always hit these.

The XHR layer is where the richest data lives. On store.example.com, opening DevTools and filtering the Network panel by Fetch/XHR typically shows calls like:

GET https://store.example.com/api/catalog?page=2
GET https://store.example.com/api/v2/products/123
GET https://store.example.com/api/reviews?product_id=123&limit=20

The catalog call returns the full product object, not the truncated version the template renders:

{
  "id": 123,
  "title": "Ultra Widget Pro",
  "pricing": { "cents": 129900, "currency": "USD", "compare_at": 149900 },
  "inventory": { "qty": 14, "warehouse": "us-east" },
  "variants": [
    { "sku": "UWP-BLK-64", "cents": 129900, "in_stock": true },
    { "sku": "UWP-BLK-256", "cents": 154900, "in_stock": false }
  ]
}

The DevTools walkthrough, in short: open the Network tab, check “Preserve log”, reload the page, filter to XHR, then click through a product listing. The request that fires when you click a product tile — the one whose response contains the full object with pricing and variants — is your endpoint. Copy it as cURL, replay it, and check whether it works without cookies. If it does, you have a data source that needs no browser at all.

Fidelity Audit: Fields the DOM Renders, Rounds, and Drops Entirely

The DOM is a lossy projection of the data. Templates decide what to show, and they show the marketing version — rounded prices, human labels, and none of the fields the business actually tracks.

Same product, both sources:

Field	DOM value	JSON value
Price	"$1,299.00" (string)	129900 + "currency": "USD"
Compare-at price	not rendered	149900
Availability	"In Stock" label	qty: 14, warehouse: "us-east"
Variant SKUs	not rendered	full array, 2 entries
Per-variant pricing	not rendered	129900, 154900
Per-variant stock	not rendered	true, false
Product ID	absent from markup	123

Every row below the first two is data the template drops. If your price-monitoring pipeline needs per-variant stock, DOM scraping cannot get it at all — not slowly, not expensively. It is simply not there.

Pulling the nested variants out of the JSON payload takes five lines:

import json, re

data = json.loads(re.search(
    r'<script id="__NEXT_DATA__"[^>]*>(.*?)</script>', html, re.S).group(1))
product = data["props"]["pageProps"]["product"]
variants = [{"sku": v["sku"], "cents": v["cents"], "in_stock": v["in_stock"]}
            for v in product["variants"]]

To make the gap concrete, run a field counter on the same page with both methods:

def count_dom_fields(soup):
    fields = {
        "title": soup.select_one("h1.pdp-title"),
        "price": soup.select_one("span.price-card__final"),
        "availability": soup.select_one("span.availability-label"),
        "rating": soup.select_one("span.rating-badge"),
        "description": soup.select_one("div.pdp-description"),
        "breadcrumbs": soup.select("nav.breadcrumb a"),
        "images": soup.select("div.gallery img"),
        "sku": soup.select_one("span.sku-label"),
        "reviews_count": soup.select_one("a.reviews-link span"),
    }
    return sum(1 for v in fields.values() if v)

def count_json_fields(product):
    def walk(node):
        if isinstance(node, dict):
            return sum(walk(v) for v in node.values())
        if isinstance(node, list):
            return sum(walk(v) for v in node)
        return 1 if node is not None else 0
    return walk(product)

# Typical result on store.example.com product pages:
# DOM: 9 extractable fields
# JSON: 31 leaf fields

Nine versus thirty-one, and the twenty-two extra fields are the ones that actually differentiate products — variant pricing, warehouse allocation, compare-at prices. This is the same reason I argue in parse at fetch time vs store raw HTML that keeping the raw payload around pays off: the DOM answer is frozen at what the template chose to show that day.

Selector Drift vs Endpoint Drift: Which Breakage Is Cheaper to Detect

Frontend teams redesign. Backend teams version. Both events break scrapers, but they break them differently, and the difference matters more than the frequency.

Here is a real-world drift pattern. Before:

<div class="price-card">
  <span class="price-card__final">$1,299.00</span>
</div>

After a redesign:

<div class="pdp-price-block">
  <span class="pdp-price--new" data-testid="price">$1,299.00</span>
</div>

The brittle version versus the endpoint version:

# DOM path: dies silently on the redesign above
price = soup.select_one("div.price-card > span.price-card__final").text
# AttributeError: 'NoneType' object has no attribute 'text' -- if you're lucky

# Endpoint path: survives the redesign untouched
r = requests.get("https://store.example.com/api/v2/products/123", timeout=5)
price = r.json()["pricing"]["cents"]

Now the comparison that actually drives the decision:

Dimension	DOM drift	Endpoint drift
Frequency	High — every visual refresh	Low — versioned paths change rarely
Failure signal	None returned, empty string, or a stale-looking default	HTTP 404, 410, or a schema key change
Detection ease	Poor — the scraper keeps “succeeding” with blanks	Good — the request fails loudly
Typical fix time	Hours — find new selectors, test, deploy	Minutes to hours — swap /v2/ for /v3/, adjust keys

The quiet failure is the dangerous one. A scraper that returns None for a price field still exits 0. It still writes rows. Your database fills with nulls for a week before anyone notices, and by then the backfill window may be gone. An endpoint that 404s fails on the first request and pages you immediately.

This is also why I push back on the common advice to “just add more fallback selectors.” Stacking selectors (span.price-card__final, span.pdp-price--new, [data-testid='price']) hides drift instead of surfacing it. You end up with a scraper that silently degrades across redesigns, and nobody knows which selector chain is live in production. Prefer one selector, fail loudly, and let a canary catch it — covered later in this post. For more on failure modes in extracted data, fix scraped data that lands in wrong fields covers the downstream symptoms.

The Failed-Probe Tax: Quantifying Latency and Crawl Budget Spent on Dead Endpoints

Tapping hidden endpoints is not free. Before you know which paths are live, you probe — and dead probes cost real time and crawl budget. Measure it before assuming the JSON path is cheaper.

Two cURL timings against store.example.com:

# Dead endpoint: internal path that no longer exists
curl -o /dev/null -s -w "http_code=%{http_code} time_total=%{time_total}s\n" \
  "https://store.example.com/api/internal/products?id=123"
# http_code=404 time_total=0.41s

# Live endpoint
curl -o /dev/null -s -w "http_code=%{http_code} time_total=%{time_total}s\n" \
  "https://store.example.com/api/v2/products/123"
# http_code=200 time_total=0.19s

A dead probe costs more than a successful hit — the server still routes, logs, and rejects it. Now multiply across a crawl. A sane probe policy looks like this:

PROBE_POLICY = {
    "max_attempts": 2,           # one retry, not five
    "timeout_ms": 2000,          # bail fast on slow probes
    "backoff_base_s": 1.5,       # exponential: 1.5s, 2.25s
    "circuit_breaker": {
        "failure_threshold": 5,  # open after 5 consecutive failures
        "cooldown_s": 3600,      # skip endpoint for an hour
    },
}

The cost math, per 10,000 pages:

Scenario	Probes/page	Failure rate	Avg dead-probe latency	Wasted time
No failure cache, 2 dead paths	2	100%	0.41s	~2.3 hours
With retry (2 attempts each)	4	100%	0.41s	~4.6 hours
Failure cache + circuit breaker	0.05	5%	0.41s	~7 minutes

Two dead paths probed twice per page burns over two hours of wall-clock time per 10k crawl — for zero data. With a circuit breaker, the same crawl wastes minutes. The failure cache in the next section is not an optimization nicety; it is the difference between a pipeline that finishes on schedule and one that does not.

One more cost people forget: dead probes look like scanning behavior to the origin server. Hammering /api/internal/* paths that return 404 is a pattern worth avoiding for reasons beyond latency. If you are routing through a scraping API, failed probes also consume your request budget — the success-based vs metered billing distinction matters here, because a metered model charges you for the 404s too.

Probe-First, DOM-Fallback: A Resilient Pipeline for store.example.com

The architecture I recommend for storefront data: try the JSON endpoint, cache failures, degrade to DOM parsing. Not the other way around — the JSON path gives higher fidelity when it works, and the DOM path keeps coverage when it does not.

import time, requests
from bs4 import BeautifulSoup

FAILURE_CACHE = {}  # endpoint -> {"last_failed_at": float, "retry_after": float}
PROBE_TIMEOUT = 2.0
CACHE_TTL = 3600  # retry a dead endpoint once per hour

def fetch_product(product_id: int) -> dict:
    endpoint = f"https://store.example.com/api/v2/products/{product_id}"

    cached = FAILURE_CACHE.get(endpoint)
    if cached and time.time() < cached["retry_after"]:
        return dom_fallback(product_id)

    try:
        r = requests.get(endpoint, timeout=PROBE_TIMEOUT,
                         headers={"Accept": "application/json"})
        if r.status_code == 200 and "pricing" in r.json():
            data = r.json()
            return {"source": "json", "price_cents": data["pricing"]["cents"],
                    "currency": data["pricing"]["currency"],
                    "variants": data.get("variants", [])}
        raise ValueError(f"unexpected shape: {r.status_code}")
    except (requests.RequestException, ValueError) as exc:
        FAILURE_CACHE[endpoint] = {
            "last_failed_at": time.time(),
            "retry_after": time.time() + CACHE_TTL,
        }
        print(f"[warn] endpoint dead ({exc}); falling back to DOM")
        return dom_fallback(product_id)

def dom_fallback(product_id: int) -> dict:
    html = requests.get(f"https://store.example.com/p/{product_id}",
                        timeout=10).text
    soup = BeautifulSoup(html, "html.parser")
    price_el = soup.select_one("span.price-card__final")
    return {"source": "dom", "price_text": price_el.text if price_el else None,
            "variants": []}  # DOM cannot supply variants

The failure cache in production should live in SQLite or Redis, not a dict — a restart should not forget which endpoints are dead:

CREATE TABLE endpoint_failures (
    endpoint     TEXT PRIMARY KEY,
    last_failed_at REAL NOT NULL,
    retry_after  REAL NOT NULL,
    reason       TEXT
);
-- INSERT INTO endpoint_failures VALUES
--   ('https://store.example.com/api/internal/products',
--    1735689600.0, 1735693200.0, 'http_404');

Log output from a real run of this pattern:

[info]  product 123 resolved via json endpoint (0.21s)
[warn]  endpoint dead (HTTPError: 404); falling back to DOM
[info]  product 124 resolved via dom fallback (1.8s)
[info]  product 125 resolved via json endpoint (0.19s)

Note the latency line: 0.21s via JSON versus 1.8s via DOM fallback. That ratio is typical when the DOM path needs JS rendering and the JSON path does not — the per-request cost difference is covered in detail in JS rendering vs plain HTTP. If you run this through a scraping API, the same trade-off applies: a plain request to the endpoint costs fewer tokens than a browser-rendered fetch of the full page. One POST /api/v1/scrape call with use_js_render: false against the endpoint, versus one with use_js_render: true against the HTML page, is the same decision expressed differently.

Canary Checks: Catching Drift on Both Paths Before Production Jobs Fail

A failure cache handles dead endpoints. A canary handles the quieter problem: endpoints and selectors that still “work” but return different data. Run a scheduled check against one known product through both paths, and assert on expected values, not just HTTP status.

canary:
  schedule: "daily 06:00 UTC"
  target:
    url: "https://store.example.com/p/123"
    endpoint: "https://store.example.com/api/v2/products/123"
  json_path_assertions:
    - path: "pricing.cents"
      equals: 129900
    - path: "pricing.currency"
      equals: "USD"
    - path: "variants[0].sku"
      equals: "UWP-BLK-64"
  dom_assertions:
    - selector: "span.price-card__final"
      contains: "1,299"
    - selector: "span.availability-label"
      contains: "In Stock"
  on_failure: "alert:#data-eng"

Alert output when things drift:

[alert] canary failed (json path): key 'pricing.cents' missing from response.
        Present keys: ['price', 'currency_code'] -- likely API schema change.
[alert] canary failed (dom path): selector 'span.price-card__final' returned None.
        Page title still 200 -- likely frontend redesign.

Two different alerts, two different causes, two different first responses:

Failure signature	Likely cause	First response
JSON keys renamed/missing, HTTP 200	API schema change or version bump	Diff response against stored sample; check for /v3/ path; update parser
HTTP 404/410 on endpoint	Endpoint removed or versioned away	Probe alternate paths; check sitemap/JS bundle for new base URL
DOM selector returns None, page loads fine	Frontend redesign	Inspect new markup; update selector; check data-testid attributes first
DOM returns stale/default values	A/B test or geo-variant serving	Compare with a second exit region; see localized storefront scraping
Both paths fail simultaneously	Site outage or anti-bot escalation	Check status page; verify with a manual fetch before touching code

The both-paths row is why the canary should hit the two paths independently. When they fail together, your problem is not drift — it is access, and no amount of parser maintenance will fix it.

When DOM Parsing Still Wins: Auth Walls, Signed Endpoints, and Rate-Limited APIs

The JSON-first recommendation has exceptions, and pretending it does not would make this article useless. Here is the big one: endpoints that require signed requests.

# The endpoint route: every request needs an HMAC signature derived from
# a per-session token, with a timestamp and nonce. Reverse-engineering
# this means extracting the signing function from a minified JS bundle,
# re-implementing it, and keeping it in sync every time the bundle changes.
headers = {
    "X-Auth-Token": session_token,
    "X-Signature": hmac.new(secret, f"{path}:{ts}:{nonce}", sha256).hexdigest(),
    "X-Timestamp": str(ts),
    "X-Nonce": nonce,
}
r = requests.get("https://store.example.com/api/orders", headers=headers)

# The DOM route: reuse the session cookies from a normal login flow,
# fetch the page the frontend already renders, parse the table.
s = requests.Session()
s.cookies.update(logged_in_cookies)
html = s.get("https://store.example.com/account/orders", timeout=10).text
rows = BeautifulSoup(html, "html.parser").select("table.orders tbody tr")

The second snippet is shorter, requires zero reverse engineering, and breaks only when the orders table markup changes — which is rare, because account pages get redesigned far less often than product pages. When the frontend guards its API with request signing, the rendered HTML is effectively a free, pre-verified projection of that same API.

The full decision matrix:

Factor	Choose JSON endpoint	Choose DOM
Rate limits	Endpoint is a first-class API, generous limits	Endpoint is aggressively throttled per session
Schema stability	Versioned, documented or stable shape	Keys churn every deploy
Auth complexity	Open or cookie-only	Signed headers, HMAC, rotating tokens
Data volume	Full objects, all variants	Template renders everything you need
Render cost	Plain HTTP fetch works	Page needs JS rendering anyway
Change cadence	Backend changes quarterly	Frontend changes quarterly but fails loudly via canary

My honest weighting: auth complexity dominates. A signed endpoint can cost a week of reverse engineering and then break monthly when the bundle rotates. A DOM parse of an authenticated page costs an afternoon and breaks maybe twice a year. Fidelity arguments do not matter if you cannot reliably make the request at all.

Wrap-Up

Scrape the source, not the projection. On a modern storefront, the rendered DOM is a lossy view of JSON the site already shipped — in hydration scripts on the first response, and in XHR endpoints the frontend calls. Reading that JSON directly gets you more fields (31 versus 9 on our example page), cleaner types (129900 + "USD" instead of "$1,299.00"), and failures that are loud instead of silent.

But the JSON path has real costs: dead probes burning hours of crawl time, schema drift, and — the deal-breaker — signed endpoints that are cheaper to avoid than to crack. The pipeline that survives is hybrid: probe the endpoint first, cache failures with a circuit breaker, fall back to the DOM, and run daily canaries on both paths so drift triggers an alert instead of a week of null values in your database.

Start by opening DevTools on your next target before writing a parser. Ten minutes of Network panel inspection will tell you which side of the decision that site falls on.