How to Scrape Dynamic Product Feeds from Storefronts

How to Scrape Dynamic Product Feeds from Hosted Storefronts in 2026

Hosted storefront platforms render product data dynamically via JavaScript. You can’t just requests.get() and expect to see the full catalog. The product list often appears after a fetch to a JSON products endpoint, but that endpoint is rate-limited, gated by a challenge page, or returns empty for unauthenticated clients. Even if you get a response, the Content-Type may be application/json while the useful payload is wrapped in a script tag or blocked with a 403. You’re not dealing with static HTML. You’re dealing with a live SPA behind access controls.

This isn’t just a scraping problem. It’s a systems engineering challenge. The data you need is real-time, but the path to it is protected. Manual inspection shows the data is there—on the client side, in React components, or in window.__cartData. But accessing it requires a browser environment, proper headers, and a clean TLS fingerprint. Even then, rate limits kick in after a handful of requests per minute.

A scrape API that combines headless browser rendering, residential proxy rotation, and challenge handling fits this case. You don’t need to manage Puppeteer instances, handle session drift, or reverse-engineer the storefront API. You make one request. The result is a structured JSON payload with product fields, images, variants, and pricing — and if the page doesn’t render, the request isn’t billed.

---

Step 1: Set Up the Request with Dynamic Rendering

The core challenge is that hosted storefronts use React and hydration to render product lists. The initial HTML contains a minimal shell. The real data is injected via window.__cartData or similar global state.

Using a simple requests.get() won’t work. Even with User-Agent spoofing, you’ll get an empty list or a redirect to a login page. You need JavaScript execution.

The use_js_render=true flag triggers a real browser to render the page. This is non-negotiable for dynamic feeds.

import requests

url = "https://store.example.com/collections/all-products"

response = requests.post(
    "https://api.finedata.ai/api/v1/scrape",
    headers={
        "x-api-key": "fd_your_api_key",
        "Content-Type": "application/json"
    },
    json={
        "url": url,
        "use_js_render": True,
        "js_wait_for": "networkidle",
        "use_antibot": True,
        "tls_profile": "chrome120",
        "timeout": 60,
        "max_retries": 3,
        "formats": ["markdown", "rawHtml"],
        "extract_rules": {
            "products": "script:contains('window.__cartData')",
            "variants": "script:contains('window.__initialState')",
        },
        "only_main_content": True
    }
)

if response.status_code == 200:
    data = response.json()
    print("Tokens used:", data["tokens_used"])
    print("Status:", data["status_code"])
    print("Page loaded in:", data["meta"]["elapsed_ms"], "ms")

This request:

• Uses use_js_render=true to run a real browser.
• Waits for networkidle—no more requests for 500ms.
• Sets tls_profile=chrome120 to match a real Chrome browser.
• Enables use_antibot so the TLS handshake matches the claimed browser.
• Sets only_main_content=true to strip navigation and ads.
• Extracts the script tag containing product data using extract_rules.

The extract_rules field is critical. It’s not only a CSS selector—it can match patterns. script:contains('window.__cartData') finds the script that contains the JSON payload.

---

Step 2: Extract Structured Product Data

Raw HTML or markdown isn’t enough. You need structured data.

extract_schema and extract_prompt let you extract only what you need. For hosted storefronts, the simplest approach is often to extract the embedded script and parse it.

{
  "url": "https://store.example.com/collections/all-products",
  "use_js_render": true,
  "js_wait_for": "networkidle",
  "use_antibot": true,
  "tls_profile": "chrome120",
  "formats": ["text"],
  "extract_rules": {
    "raw_json": "script:contains('window.__cartData')",
    "products": "script:contains('window.__cartData')",
    "variants": "script:contains('window.__initialState')"
  },
  "extract_schema": {
    "type": "object",
    "properties": {
      "products": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "title": { "type": "string" },
            "price": { "type": "number" },
            "compare_at_price": { "type": "number", "nullable": true },
            "image": { "type": "string", "format": "uri" },
            "handle": { "type": "string" },
            "variants": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "title": { "type": "string" },
                  "price": { "type": "number" },
                  "sku": { "type": "string" }
                }
              }
            }
          }
        }
      }
    }
  }
}

This extract_schema tells the model to look for product objects. The model parses the script content, extracts the JSON, and returns a clean object.

The response includes:

{
  "success": true,
  "status_code": 200,
  "data": {
    "text": "window.__cartData = { ... }",
    "products": [
      {
        "title": "Organic Cotton T-Shirt",
        "price": 24.99,
        "compare_at_price": 29.99,
        "image": "https://cdn.example.com/s/files/1/0000/0000/products/tshirt.jpg?v=1680000000",
        "handle": "organic-tshirt",
        "variants": [
          {
            "title": "Black, Large",
            "price": 24.99,
            "sku": "TSHIRT-BLK-L"
          }
        ]
      }
    ]
  },
  "tokens_used": 12,
  "meta": {
    "url": "https://store.example.com/collections/all-products",
    "resolved_url": "https://store.example.com/collections/all-products",
    "elapsed_ms": 3120,
    "proxy_country": "US"
  }
}

You get structured data. No regex. No brittle parsing. The model handles nested structures, missing fields, and malformed JSON.

Pro tip: Use ai_content_mode=full if you want the model to see the full page, including sidebars. Use ai_content_mode=main if you want it to ignore navigation and focus on the product list. Prefer main when the catalog is the only signal you need—it’s faster and avoids noise.

---

Step 3: Handle Rate Limits and Session Persistence

Even with a matching TLS profile, storefront rate limits kick in after ~10–15 requests per minute per IP. You’ll see 429s or blocked responses.

session_id and session_ttl help here.

{
  "url": "https://store.example.com/collections/all-products",
  "use_js_render": true,
  "js_wait_for": "networkidle",
  "use_residential": true,
  "session_id": "storefront-feed-001",
  "session_ttl": 1800,
  "use_antibot": true,
  "tls_profile": "vip:ios",
  "formats": ["json"],
  "extract_schema": { ... }
}

This request:

• Uses a residential proxy (use_residential=true) when datacenter exits fail often.
• Sets session_id to reuse the same IP for 30 minutes.
• Uses vip:ios TLS profile—emulates an iPhone browser with an iOS fingerprint.

The result? You can make many more requests per session before rate limiting. The proxy pool is shared across users, but the session keeps the same IP.

Trade-off: Residential proxies cost 3 tokens per request. But they’re often worth it. In practice, 403 rates drop sharply with this setup — still probabilistic, and failed renders aren’t billed.

---

Step 4: Build a Batch Job for Large Feeds

For stores with 10,000+ products, you need to paginate. Many hosted storefronts use ?page=2, ?limit=50, etc.

Use the /api/v1/async/batch endpoint to submit 100+ URLs at once.

batch_data = {
  "requests": [
    {
      "url": "https://store.example.com/collections/all-products?page=1&limit=50",
      "use_js_render": true,
      "js_wait_for": "networkidle",
      "use_residential": true,
      "session_id": "storefront-batch-001",
      "session_ttl": 3600,
      "formats": ["json"],
      "extract_schema": { ... }
    },
    {
      "url": "https://store.example.com/collections/all-products?page=2&limit=50",
      "use_js_render": true,
      "js_wait_for": "networkidle",
      "use_residential": true,
      "session_id": "storefront-batch-001",
      "session_ttl": 3600,
      "formats": ["json"],
      "extract_schema": { ... }
    }
  ],
  "callback_url": "https://your-webhook.com/finedata/callback",
  "timeout": 120
}

response = requests.post(
    "https://api.finedata.ai/api/v1/async/batch",
    headers={"x-api-key": "fd_your_api_key"},
    json=batch_data
)

batch_id = response.json()["batch_id"]
print("Batch submitted:", batch_id)

The webhook will send a POST when all jobs complete. You can then merge the results.

Why batch? It’s more efficient than polling. You don’t need to check 100 jobs individually. The API returns a single batch_id and a final status.

---

Gotchas and Trade-Offs

1. extract_schema is not a parser. It’s an LLM prompt. If the script is minified or uses obfuscation, it might fail. Test with rawHtml first.

2. vip:ios and vip:android are expensive—15 tokens per request. They are among the profiles with the highest success rate on pages that gate content behind a challenge. If you’re scraping 100 stores, the cost can be justified by fewer retries.

3. js_wait_for=networkidle is not always reliable. Some storefronts use WebSockets or infinite polling. Use selector:.product-card to wait for a visible product.

4. Dynamic storefronts usually need a browser. Even if you find a JSON endpoint, it often returns 403 unless you send a consistent browser fingerprint. The API handles that—your app doesn’t need to.

5. Residential proxies are not anonymous. They’re real devices. But they’re not tied to your IP. The residual risk is low, not zero. Use session_id to reduce unnecessary IP churn.

6. Prefer extract_schema over extract_prompt. It’s more predictable. extract_prompt is like asking a model to “extract all products.” It works, but you get inconsistent output. extract_schema is more deterministic.

---

Next Steps

1. Build a scheduler. Use session_id to make dozens of requests per sticky window. Fewer rate-limit hits.

2. Add caching. Store the last updated_at timestamp. Only re-scrape if the product list changed.

3. Use MCP. Connect your AI agent to the scraped data. MCP Protocol: How to Connect AI Agents to Web Data lets you build agents that monitor storefront catalogs in real time.

4. Add error monitoring. Track failed jobs. Use GET /api/v1/async/jobs to check status.

5. Scale to 100 stores. Use batch jobs. Use callback_url to avoid polling.

---

Final Thoughts

Scraping hosted storefront product feeds isn’t about writing clever regex or managing Puppeteer clusters. It’s about choosing the right tools.

A managed scrape API abstracts away:

• Browser-grade request fidelity
• Proxy rotation
• JavaScript rendering
• Inline challenge handling
• Structured extraction

You don’t need to write a scraper. You write a data pipeline.

The real win isn’t speed. It’s reliability. With a session_id and a strong TLS profile, you get more consistent access. Fewer 403s. Fewer IP bans. Still probabilistic — and you only pay for successful renders.

If you’re building a price monitor, a catalog sync, or a product intelligence platform, this is the stack you want.

Before pointing this at a live store, read its terms of service and robots.txt, and keep to the public catalog — customer data, order flows and anything behind a login are out of scope, and our acceptable use policy says the same. The legal guide covers where the lines actually fall.

Just don’t do it with bare requests. Use a browser-grade API — FineData if you want one call instead of a proxy-and-browser fleet.