How to Scrape Marketplace Product Data with Python

How to Scrape Marketplace Product Data with Python

Large retail marketplaces hold tens of millions of product listings, and the data on those pages — price, availability, rating, review count, variation matrix — feeds price comparison tools, competitive research, and analytics pipelines. Getting it programmatically is one of the harder scraping problems, and not because the HTML is complicated.

Two approaches follow below: fetching pages and parsing the HTML yourself, and having the API return the fields already structured. They cost different amounts and break in different ways, so the second half of the article is mostly about which one to reach for.

Why marketplace product pages resist simple scripts

A plain requests.get() against a major marketplace usually runs into some combination of:

• Dynamic content — price, stock and reviews arrive via JavaScript after the initial HTML.
• Request fingerprinting — the TLS handshake, header order and browser characteristics are inspected together, so a Python HTTP client looks nothing like Chrome even with a copied user-agent.
• IP reputation scoring — datacenter ranges get flagged quickly on high-traffic product IDs.
• Challenge pages — served on behaviour patterns rather than on volume alone.
• Rate limiting — even with rotating exits, bursts get throttled.

The pattern most teams hit is the same: requests plus BeautifulSoup works for a dozen pages, then the 403s start, retries get added, a headless browser gets bolted on, and it works locally but not in production. The expensive part is not compute — it is the engineering time spent chasing a fingerprint that keeps moving.

Setting up

pip install requests beautifulsoup4

You will also need an API key — sign up at finedata.ai and copy it from the dashboard.

import requests
from bs4 import BeautifulSoup
import json
import time

API_KEY = "fd_your_api_key"
SCRAPE_URL = "https://api.finedata.ai/api/v1/scrape"

def scrape_page(url, use_js=False):
    """Fetch a page through the scrape API."""
    response = requests.post(
        SCRAPE_URL,
        headers={
            "x-api-key": API_KEY,
            "Content-Type": "application/json"
        },
        json={
            "url": url,
            "use_js_render": use_js,
            "tls_profile": "chrome124",
            "use_residential": True,
            "timeout": 60
        }
    )
    response.raise_for_status()
    return response.json()

Three parameters are doing the work here. tls_profile makes the handshake match a real Chrome build, so the transport layer is consistent with what the request claims to be. use_residential routes through consumer ISP addresses, which matters on marketplaces that score datacenter ranges harshly. use_js_render runs the page in a real browser — necessary whenever pricing is injected client-side.

timeout: 60 is not padding. Marketplace pages with heavy client-side rendering routinely take 15–20 seconds; a 30-second ceiling turns normal pages into gateway timeouts.

Parsing the page yourself

The classic approach: get the HTML, hand it to BeautifulSoup, pull out fields. Selectors differ per site, so treat the ones below as a shape rather than something to copy verbatim.

def parse_product_page(html):
    """Extract product details from a marketplace product page.

    Selectors are examples; adjust them per target site.
    """
    soup = BeautifulSoup(html, "html.parser")
    product = {}

    title_el = soup.select_one("h1#product-title, h1.product-title, h1")
    product["title"] = title_el.get_text(strip=True) if title_el else None

    # Retail sites often carry several price containers for the same number
    price_el = (
        soup.select_one(".price .amount")
        or soup.select_one("[data-price]")
        or soup.select_one(".product-price")
        or soup.select_one(".sale-price")
    )
    product["price"] = price_el.get_text(strip=True) if price_el else None

    rating_el = soup.select_one(".rating, [itemprop='ratingValue'], .star-rating")
    if rating_el:
        try:
            product["rating"] = float(rating_el.get_text(strip=True).split(" ")[0])
        except ValueError:
            product["rating"] = None
    else:
        product["rating"] = None

    reviews_el = soup.select_one(".review-count, [data-review-count]")
    if reviews_el:
        digits = reviews_el.get_text(strip=True).split(" ")[0].replace(",", "")
        product["review_count"] = int(digits) if digits.isdigit() else None
    else:
        product["review_count"] = None

    avail_el = soup.select_one(".availability, [data-availability]")
    product["availability"] = avail_el.get_text(strip=True) if avail_el else None

    bullets = soup.select(".feature-list li, .product-features li")
    product["features"] = [b.get_text(strip=True) for b in bullets if b.get_text(strip=True)]

    return product


def scrape_product(product_id):
    url = f"https://store.example.com/product/{product_id}"
    result = scrape_page(url, use_js=True)

    product = parse_product_page(result["body"])
    product["product_id"] = product_id
    product["url"] = url
    return product


product = scrape_product("SKU-48219")
print(json.dumps(product, indent=2))

Every field here is written to degrade to None rather than raise. That is deliberate: marketplace listings are not uniform. Some carry several sellers at different prices, some have subscription pricing, some are out of stock with no price element at all. A parser that assumes every element exists will die on the first unusual listing in a batch of a thousand.

Product variations

Size, colour and configuration variants are usually loaded on click, with the underlying map embedded in a JavaScript object in the page source. Extracting it saves a request per variant:

import re

def extract_variations(html):
    """Pull the variation map out of the page source.

    Key names differ per site; this shows one common embedding style.
    """
    variations = []
    pattern = r'"variationMap"\s*:\s*(\{[^}]+\})'
    match = re.search(pattern, html)

    if match:
        try:
            dim_data = json.loads(match.group(1))
            for sku, values in dim_data.items():
                variations.append({"sku": sku, "attributes": values})
        except json.JSONDecodeError:
            pass

    return variations

Finding products without a list of IDs

When you do not have product IDs to start from, catalog search is the entry point:

def scrape_catalog_search(query, max_pages=3):
    """Walk catalog search results for a query."""
    all_products = []

    for page in range(1, max_pages + 1):
        url = (
            f"https://store.example.com/search"
            f"?q={query.replace(' ', '+')}&page={page}"
        )
        result = scrape_page(url, use_js=True)
        soup = BeautifulSoup(result["body"], "html.parser")

        for item in soup.select(".search-result, .product-card, [data-product-id]"):
            product = {"product_id": item.get("data-product-id") or ""}

            title_el = item.select_one("h2 a span, .product-title, h2")
            product["title"] = title_el.get_text(strip=True) if title_el else None

            price_whole = item.select_one(".price-whole, .price .whole")
            price_frac = item.select_one(".price-fraction, .price .fraction")
            if price_whole:
                price_str = price_whole.get_text(strip=True).rstrip(".")
                if price_frac:
                    price_str += "." + price_frac.get_text(strip=True)
                product["price"] = float(price_str.replace(",", ""))
            else:
                product["price"] = None

            product["url"] = (
                f"https://store.example.com/product/{product['product_id']}"
            )
            all_products.append(product)

        time.sleep(2)

    return all_products


results = scrape_catalog_search("wireless earbuds", max_pages=2)
print(f"Found {len(results)} products")

Letting the API return structured fields

Everything above assumes you own the parsing. You do not have to. extract_rules moves the selectors into the request, and the response comes back as JSON:

payload = {
    "url": "https://store.example.com/product/SKU-48219",
    "use_antibot": True,
    "tls_profile": "chrome124",
    "use_js_render": True,
    "js_wait_for": "networkidle",
    "solve_captcha": True,
    "formats": ["markdown"],
    "extract_rules": {
        "title": "h1.product-title",
        "price": "span.price",
        "rating": "span.rating",
        "description": "#product-description"
    },
    "timeout": 60,
    "max_retries": 3
}

response = requests.post(SCRAPE_URL, json=payload, headers={
    "x-api-key": API_KEY,
    "Content-Type": "application/json"
})

data = response.json()
if not data.get("success"):
    print("Scrape failed:", data.get("error"))
else:
    print(json.dumps(data["data"], indent=2))

{
  "markdown": "### **Wireless Earbuds Pro (2nd Generation)**\n\n**Price:** $249.00\n\n**Customer Reviews:** 4.7 out of 5 stars\n",
  "text": "Wireless Earbuds Pro (2nd Generation)\n\nPrice: $249.00\n\nCustomer Reviews: 4.7 out of 5 stars",
  "links": [],
  "screenshot": "https://cdn.finedata.ai/screenshot/abc123.png"
}

js_wait_for: "networkidle" is what keeps the price field from coming back empty — it holds until the network settles instead of grabbing the first paint. solve_captcha handles an inline challenge if one appears, and a request that never renders is not billed, so a page you failed to read costs nothing.

When CSS rules are the wrong tool

extract_rules is cheap and predictable, and it breaks the moment a class name changes — which marketplaces do on their own schedule, without telling you. extract_schema describes the shape of the data instead of its location:

"extract_schema": {
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "price": { "type": "number" },
    "rating": { "type": "number" },
    "description": { "type": "string" }
  },
  "required": ["title", "price"]
}

That survives a class rename, because “$249.00” is still recognisably a price regardless of what wraps it. The trade-off is 5 extra tokens per request. For a scraper that runs unattended for months, the maintenance it removes is usually worth more than the tokens it costs — which is the same argument for AI-assisted extraction more generally.

One thing not to reach for on product pages: only_main_content: true. Complex retail layouts put price and title outside whatever the readability heuristic decides is the main block, so it happily strips the exact fields you came for.

Scaling past a few hundred pages

Sequential requests stop being viable somewhere in the low hundreds. The batch endpoint parallelises the fetching:

def scrape_products_batch(product_ids, batch_size=20):
    all_products = []

    for i in range(0, len(product_ids), batch_size):
        urls = [
            f"https://store.example.com/product/{pid}"
            for pid in product_ids[i:i + batch_size]
        ]

        response = requests.post(
            "https://api.finedata.ai/api/v1/batch",
            headers={"x-api-key": API_KEY, "Content-Type": "application/json"},
            json={"urls": urls, "use_js_render": True, "use_residential": True}
        )
        batch_id = response.json()["batch_id"]

        while True:
            status = requests.get(
                f"https://api.finedata.ai/api/v1/batch/{batch_id}",
                headers={"x-api-key": API_KEY}
            ).json()

            if status["status"] == "completed":
                for job in status["results"]:
                    if job["status"] == "completed":
                        product = parse_product_page(job["body"])
                        product["url"] = job["url"]
                        all_products.append(product)
                break

            time.sleep(5)

    return all_products

For jobs that run longer than a request should wait, POST /api/v1/async/scrape and POST /api/v1/async/batch take a callback_url and push results to your webhook when they finish. Two parameters matter at this size: session_id keeps the same exit IP across related requests, which stops a multi-step flow from looking like several unrelated visitors, and session_ttl controls how long that IP is held.

Rate discipline still applies. Rotating residential exits do not make hundreds of requests per second acceptable — pace sequential work at 1–3 seconds per request, or hand the pacing to the batch endpoints. And cache: product data does not move second to second, so a 15–30 minute cache removes a large share of requests for free.

What it costs

Operation	Tokens	Notes
Base request	1	Charged on success only
JS rendering	+5	Needed for client-side pricing
Residential proxy	+3	Recommended on large marketplaces
Schema extraction	+5	Optional, replaces CSS rules
Typical product page	9–14

A thousand product pages lands between 9,000 and 14,000 tokens depending on whether you extract by schema. Challenge solves add 10 tokens each. Requests that fail to render are not billed at all, which is what makes the arithmetic predictable — you are paying per page you actually received.

Storing the results

import sqlite3

def init_db():
    conn = sqlite3.connect("marketplace_products.db")
    conn.execute("""
        CREATE TABLE IF NOT EXISTS products (
            product_id TEXT PRIMARY KEY,
            title TEXT,
            price REAL,
            rating REAL,
            review_count INTEGER,
            availability TEXT,
            features TEXT,
            scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.commit()
    return conn


def save_product(conn, product):
    conn.execute("""
        INSERT OR REPLACE INTO products
        (product_id, title, price, rating, review_count, availability, features)
        VALUES (?, ?, ?, ?, ?, ?, ?)
    """, (
        product.get("product_id"),
        product.get("title"),
        product.get("price"),
        product.get("rating"),
        product.get("review_count"),
        product.get("availability"),
        json.dumps(product.get("features", []))
    ))
    conn.commit()

INSERT OR REPLACE keyed on product_id gives you idempotent re-runs. Once you want price history rather than current state, that schema needs a timestamped row per observation instead — which is the subject of building a price monitoring tool.

Before you point this at a real site

Check the target’s terms of service and robots.txt, and keep to publicly accessible pages. Product data behind a login, checkout flows, and anything covered by a site’s own API terms are outside what this pipeline is for. Our acceptable use policy draws the same line, and the legal guide covers the case law in more detail.

Summary

• Marketplace product pages are hard because of fingerprinting, IP scoring and client-side rendering — not because the HTML is difficult.
• Parse the HTML yourself when you need fields no generic extractor would know about; let extract_rules or extract_schema do it when the fields are ordinary.
• CSS rules are cheaper and break on class renames; schema extraction costs 5 tokens more and survives them.
• Write every parser to degrade to None. Listings are not uniform, and one unusual product should not kill a batch.
• Use batch and async endpoints past a few hundred pages, and session_id when several requests belong to one flow.
• timeout: 60, js_wait_for: "networkidle", and avoiding only_main_content remove most of the empty-field bugs.

Prototyping first? Sign up for free tokens and see what the free tier actually includes. For the anti-detection side of the same problem, see writing a Python scraper that stays consistent, the guide to inline challenges, or the API documentation.