Web Scraper in Python: A Production Build with the FineData API

Web Scraper in Python: A Production Build with the FineData API

You’re not here for another requests-based scraper that fails on the third request. You’re building something that survives in production: a web scraper in Python that gets past JavaScript rendering and anti-bot checks and returns structured data — without your infrastructure becoming a maintenance liability.

This isn’t magic. It’s engineering.

Rate-limiting, fingerprinting, and bot detection are more aggressive than they used to be, and they keep changing. The old stack — requests + BeautifulSoup + Selenium — is a maintenance nightmare. You’re either rate-limited, blocked, or spending hours debugging why playwright crashes on a single page. The cost of ownership for a DIY scraper has never been higher — see the full cost breakdown if you want the numbers.

FineData’s API won’t fix a bad scraping strategy on its own. But it’s a production-grade web scraping layer you can plug into your pipeline without writing a single line of browser automation.

Let’s build a scraper that actually works.

---

Why Plain requests Gets Detected

A year ago, I ran a scraper on 100+ e-commerce sites. I used playwright with requests to fetch pages, BeautifulSoup to extract data, and a proxy rotation layer. It worked for 3 days.

Then one target started returning 403 with a jschl-v-style challenge. Not just once. Every 12–15 requests. I spent 40 hours trying to patch around the JS challenge, only to have it break again in 2 weeks. That’s the actual lesson: patching around a specific challenge is a losing, recurring cost, not a one-time fix.

The problem isn’t just JavaScript. Plain HTTP clients get flagged by TLS fingerprinting before a single line of your scraping logic runs — the handshake itself gives away that you’re not a browser. On top of that there’s user-agent consistency, session fingerprinting, and behavioral analysis.

You can’t fake a real browser with playwright alone. You need:

• Real TLS fingerprints (Chrome, Firefox, Safari)
• Residential or mobile proxy rotation
• Consistent browser rendering for sites that present challenge pages
• CAPTCHA handling for the cases you can’t avoid
• JavaScript rendering
• Structured data extraction

And yes — you need a system that survives a 10k-page crawl.

---

The Architecture: An Anti-Bot Abstraction Layer

Instead of maintaining a fleet of Playwright instances, I now use a managed scrape API as a single, reliable abstraction layer.

The API handles the parts that make a scraper look like a browser instead of a script:

• TLS fingerprint matching (Chrome, Firefox, Safari profiles)
• Stealth rendering (Playwright and Patchright profiles)
• Residential and mobile proxy rotation
• CAPTCHA handling (reCAPTCHA, hCaptcha, Turnstile) when a challenge can’t be avoided
• JavaScript rendering
• Browser-grade request consistency for sites that gate content behind challenge pages

You send a single POST request. It returns HTML, Markdown, text, or structured JSON.

No more debugging why page.evaluate() failed because of a missing __ow function.

No more spending 3 hours on a navigator.webdriver check that’s not even in the DOM.

---

Step 1: Set Up Your Python Environment

# requirements.txt
httpx==0.24.0
pydantic==2.4.0
python-dotenv==1.0.0

Create a .env file:

FINE_DATA_API_KEY=fd_your_api_key

Use httpx for async HTTP calls. It’s faster than requests, supports streaming, and integrates cleanly with async/await.

# scraper.py
import httpx
from pydantic import BaseModel
from dotenv import load_dotenv
import os

load_dotenv()

class ScrapedProduct(BaseModel):
    title: str
    price: float
    rating: float | None = None
    description: str | None = None

class FineDataClient:
    def __init__(self):
        self.api_key = os.getenv("FINE_DATA_API_KEY")
        self.base_url = "https://api.finedata.ai"
        self.headers = {
            "x-api-key": self.api_key,
            "Content-Type": "application/json",
        }

    async def scrape(self, url: str, extract_prompt: str | None = None):
        payload = {
            "url": url,
            "formats": ["html", "text"],
            "use_js_render": True,
            "stealth_antibot": True,
            "use_residential": True,
            "extract_prompt": extract_prompt or "Extract title, price, rating, and description as JSON.",
            "only_main_content": True,
        }

        try:
            response = await httpx.post(
                f"{self.base_url}/api/v1/scrape",
                json=payload,
                headers=self.headers,
                timeout=30.0,
            )
            response.raise_for_status()
            return response.json()
        except httpx.HTTPStatusError as e:
            print(f"HTTP error: {e.response.status_code} - {e.response.text}")
            return None
        except httpx.RequestError as e:
            print(f"Request error: {e}")
            return None

Use httpx over requests for async. The performance difference is measurable at scale.

Test the API endpoint with curl first. If you get a 500 error, it’s likely a malformed payload or invalid API key. Check the API documentation for the full request schema.

---

Step 2: Handle Challenge Pages with Stealth Mode

Modern anti-bot challenges check more than a single header. They look at:

• JavaScript execution
• DOM mutation detection
• User-agent and header consistency
• Mouse movement simulation (in some cases)

The stealth_antibot: true flag enables:

• Real browser fingerprinting (recent Chrome profile)
• Headless browser with a stealth-patched Playwright profile
• Proxy rotation

async def scrape_with_stealth_mode(self, url: str):
    payload = {
        "url": url,
        "formats": ["html", "text"],
        "use_js_render": True,
        "stealth_antibot": True,
        "use_residential": True,
        "extract_prompt": "Extract title, price, rating, and description. Return as JSON.",
        "only_main_content": True,
    }

    response = await self.client.post(f"{self.base_url}/api/v1/scrape", json=payload, headers=self.headers)
    return response.json()

This is the mode to try on e-commerce sites that gate JavaScript-heavy pages behind a challenge. If the page still does not render, the request is not billed.

Trade-off: Residential proxies are slower than data center ones. But if you’re scraping sites with aggressive bot detection, the 300–500ms delay is usually worth it.

---

Step 3: Reduce How Often You See a CAPTCHA

reCAPTCHA, hCaptcha, and Cloudflare Turnstile all key off the same underlying risk signals: IP reputation, TLS fingerprint, and request pacing. The most effective “handling” strategy is not seeing the challenge at all — get those three signals right and most targets never show one:

async def scrape_with_low_captcha_risk(self, url: str):
    payload = {
        "url": url,
        "formats": ["html", "text"],
        "use_js_render": True,
        "stealth_antibot": True,
        "use_residential": True,
        "extract_prompt": "Extract product title, price, and rating. Return as JSON.",
        "only_main_content": True,
    }

    response = await self.client.post(f"{self.base_url}/api/v1/scrape", json=payload, headers=self.headers)
    return response.json()

For the residual cases where a challenge still appears, solve_captcha: true is available as a fallback — see the full CAPTCHA-handling guide for when it’s worth the added latency and token cost versus fixing the underlying signal.

---

Step 4: Extract Structured Data with LLM-Powered Prompting

This is where the real power lies. You’re not just scraping HTML. You’re extracting structured data.

The API supports LLM-powered structured extraction. You provide a prompt. It returns a JSON object.

extract_prompt = """
Extract the following from the HTML:
- title: product title
- price: numeric value in USD
- rating: float between 0.0 and 5.0
- description: short summary of features

Return only valid JSON. Do not include markdown or code blocks.
"""

async def scrape_product(self, url: str):
    payload = {
        "url": url,
        "formats": ["json"],
        "use_js_render": True,
        "stealth_antibot": True,
        "use_residential": True,
        "extract_prompt": extract_prompt,
        "only_main_content": True,
    }

    response = await self.client.post(f"{self.base_url}/api/v1/scrape", json=payload, headers=self.headers)
    return response.json()

Example response:

{
  "title": "Sony WH-1000XM5 Wireless Headphones",
  "price": 279.99,
  "rating": 4.8,
  "description": "Over-ear noise-cancelling headphones with 30-hour battery life and AI voice pickup."
}

This is not prompt engineering. It’s prompt design. You’re not training a model. You’re describing the output format clearly.

Pro tip: Use pydantic to validate the response. It’s faster and safer than json.loads() with string parsing.

---

Step 5: Scale to 10k Pages with Async Batching

For large-scale jobs, use the async batch endpoint.

async def scrape_batch(self, urls: list[str], extract_prompt: str):
    payload = {
        "urls": urls,
        "formats": ["json"],
        "use_js_render": True,
        "stealth_antibot": True,
        "use_residential": True,
        "extract_prompt": extract_prompt,
        "only_main_content": True,
    }

    response = await self.client.post(f"{self.base_url}/api/v1/async/batch", json=payload, headers=self.headers)
    return response.json()

Max 100 URLs per batch. Use asyncio.gather() to parallelize.

Don’t send 10k URLs at once. Use a queue (e.g., aiosqlite or aiopg) and throttle requests to avoid rate-limiting.

---

Real-World Example: A Product Page Scraper

# Example: product page scraper, reusable across storefronts
async def scrape_product_page(self, product_url: str):
    extract_prompt = """
    Extract:
    - title: product title
    - price: numeric value in USD (e.g. 29.99)
    - rating: float between 0.0 and 5.0
    - review_count: integer number of reviews
    - features: list of 3–5 key product features

    Return only valid JSON. Do not include markdown or code blocks.
    """

    return await self.scrape(product_url, extract_prompt)

The same function works across product pages from different storefronts — the extraction logic is driven by the prompt, not by site-specific CSS selectors, so you’re not maintaining a separate parser per target. No more selenium sessions or playwright timeouts.

If a large marketplace catalog is your target, see the dedicated guide to marketplace product data for pagination and catalog-crawl patterns this snippet doesn’t cover.

---

Trade-Offs and Real Talk

You’re not fighting AI directly. You’re dealing with detection systems that use AI to score your traffic — the same risk signals (IP reputation, TLS fingerprint, behavior) matter regardless of what’s scoring them.

• Cost: A managed scrape API is not free. But it’s cheaper than maintaining a proxy farm, CAPTCHA solvers, and browser clusters.
• Latency: 2–4 seconds per request. Acceptable for batch jobs. Not for real-time.
• Rate Limits: 100 requests/minute per API key. Use asyncio.sleep(1) between batches.
• Data Quality: The LLM extraction is good, but not perfect. Validate with a small sample.

Use only_main_content: true to reduce payload size and improve extraction accuracy.

Don’t use use_residential: true on low-traffic sites. It’s overkill.

Never hardcode your API key. Use environment variables.

---

Final Thoughts

A web scraper in Python isn’t about requests or BeautifulSoup. It’s about resilience.

At scale, the only sustainable path is to offload anti-bot handling to a managed system rather than patching your own client every time a target updates its detection — see how Cloudflare, DataDome, and PerimeterX actually detect bots for what’s actually being checked.

FineData isn’t a replacement for your logic. It’s a reliability layer.

You still write the data pipeline. You still validate the output. You still store the results.

But you don’t spend 40 hours reverse-engineering a jschl-v challenge.

You don’t debug why navigator.webdriver is true in a Playwright session.

You don’t pay $100/month for a CAPTCHA solver.

You don’t have to worry about TLS fingerprinting.

You just call the API.

And it works.

For more on how AI is reshaping data extraction, see The Future of Web Scraping: AI, LLMs, and Structured Extraction.

---

Ready to Build?

Set up your API key, write a single httpx call, and you’re live.

No more 403 errors. No more jschl-v puzzles. No more navigator.webdriver bugs.

Just data.

And that’s what matters.