Tutorials
Step-by-step web scraping tutorials with working Python and API examples.
Your Scraper Died With Exit Code 137: Fix the OOM Kill
Workers die mid-batch with exit code 137 and no logged exception. Learn to confirm the OOM kill, find the memory hog, and stop buffering full page bodies.
Batch a URL List Into Partitioned Parquet Files
Turn a list of URLs into partitioned Parquet files with a staged Python pipeline: fetch, extract, validate, and load, with a run manifest you can audit.
Your Backoff Ignores Retry-After: Fix Persistent 429s
Still throttled despite exponential backoff? Learn to parse Retry-After headers, cap per-host concurrency, and stop the 429 cycle stalling your scrape pipeline.
Too Many Redirects: Fix Redirect Loops in Your Scraper
Your scraper dies with 'too many redirects' on a page that loads fine in a browser. Trace the cookie and geo causes, then cap and fix the loop.
Backfill, Then Refresh: A Two-Phase Scrape Pipeline
Build a scrape pipeline that backfills once, then refreshes on a schedule. Working Python code, checkpoints, and a job queue you can run today.
Certificate Verify Failed: Fix SSL Errors in Web Scrapers
Your scraper throws SSLError: certificate verify failed. Learn the real causes — TLS inspection, stale CAs — and fix verification without turning it off.
Your Scraped Timestamps Say '2 Days Ago': Fix Relative Dates
Scraped pages return '2 hours ago' instead of real datetimes. Learn why relative dates break time-series data and how to normalize them at parse time.
Too Many Open Files: Fix a Socket Leak in Your Scraper
Scrape workers crash with 'too many open files' mid-batch. Trace the socket leak, confirm it with lsof, and fix it with pooled connections and closed responses.
Trip a Circuit Breaker When Your Scrape Target Degrades
Success rate tanks mid-run and retries pile up. Learn to separate target-site degradation from your own bugs, then trip a circuit breaker and fail fast.
Turn a Batch of Scraped Pages into a Queryable Dataset
Build the transform and load stages of a scraping pipeline: normalize page records, dedupe on upsert, and land them in SQLite with working code.
Your Scraped Links Don't Resolve: Fix Relative URLs
Scraped hrefs like /products/42 wreck downstream crawls and dedupe keys. Learn how to resolve relative URLs to absolute ones before you store them.
Your 200 OK Responses Are Actually Block Pages
A 200 status code doesn't mean you got the page. Detect challenge pages and interstitials hiding inside successful responses, then fix your pipeline.
Build a Scraping Pipeline You Can Safely Restart Mid-Run
A tutorial on checkpointing batch scraping runs with run manifests and idempotent writes, so a crashed pipeline resumes exactly where it stopped.
Garbled Scraped Text: Fix Character Encoding Before Parsing
Your scraper returns mojibake instead of product names. Trace the failure from missing charset headers to wrong decode steps and fix it before parsing.
Throughput Decay Before the 429s: Fix Hidden Throttling
Your batch pace slows long before requests fail. Learn to tell target-site throttling from your own concurrency limits and fix it with adaptive pacing.
Re-Drive Only the Failures: Handling Partial Batch Results
A batch scrape rarely fails cleanly. Learn to reconcile partial results, classify why each URL failed, and re-drive only the misses without paying twice.
Scrape Mobile Pages Through a Carrier Exit IP
Some sites serve challenges to desktop traffic but real pages to phones. Learn when a mobile carrier exit IP, set with use_mobile, is the fix.
Your Scraper Died in a Redesign: An Extraction Escalation
A site redesign turned your working selectors into nulls overnight. Fix it with an escalation ladder: CSS rules, then schema, then prompt extraction.
Fix Scraped Data That Lands in the Wrong Fields
Extracted JSON looks valid, but prices land in the wrong field. Trace the cause and fix mis-mapped data with CSS selectors, a JSON schema, or a prompt.
Stop Cleaning HTML by Hand: Scrape Pages as Markdown
Chunking raw HTML for LLM pipelines wastes tokens and breaks parsers. See how one scrape request field returns clean, ready-to-chunk markdown instead.
Scrape Localized Storefronts With a Country Exit Code
Storefronts change price, language, and stock by visitor country. Pin the scrape exit with an ISO-2 code and handle 422 when the country is unsupported.
Route Scraping Traffic Through Proxies You Already Own
When targets allowlist your IPs or you already pay for residential proxies, attach a proxy profile so scrape requests exit through your pool.
Steam 429 Error: Parsing Store Pages Responsibly with Python
Handle Steam 429 errors without retrying early: respect Retry-After, separate API and target failures, and extract Portal 2 store data with Python.
Wikipedia Scraping Returns None: Fix the Page and the Selector
Fix Wikipedia scraping that returns None: distinguish failed retrieval from Parsoid selector changes, validate extracted fields, and preserve text spacing.
Keep One Exit IP Across Multi-Step Scrape Requests
Learn how proxy_sticky keeps the same exit IP for warmup and follow-up requests, preventing session resets during multi-step scraping workflows.
Sticky Exit IPs for Warmup and Follow-Up Scrapes
Rotating proxies between a warmup hit and the next request breaks cookies and geo checks. Learn how a sticky exit IP keeps the session on the same IP.
Async Scraping at Scale: Jobs, Batches, Webhooks
Practical guide to FineData async scraping: submit jobs, poll with exponential backoff, verify webhook callbacks, and run batches of up to 100 URLs.
From HTML to JSON: Schema-Driven Extraction
Turn scraped pages into validated JSON with FineData: pick extract_schema, extract_prompt, or extract_rules, tune ai_content_mode, and export CSV/XLSX.
Free No-Code Web Scraper: Extract Data Without Writing Code
How to use no-code web scrapers to extract structured data from websites. Tools, workflows, and practical limitations for non-developers.
How to Scrape Job Postings with Dynamic Filters Using FineData API
Step-by-step guide to extract job listings from career sites with dynamic filters using FineData's API and Playwright rendering.
Web Scraper in Python: A Production Build with the FineData API
Build a production-grade Python web scraper that handles JavaScript rendering, anti-bot systems, and structured extraction using FineData's API, with working code examples.
How to Scrape Dynamic Product Feeds from Storefronts
Extract real-time product data from hosted storefronts that render their catalog in JavaScript — browser rendering, sticky sessions, batch jobs.
Building a Price Monitoring Tool: Step-by-Step Guide
Build a complete price monitoring tool with Python. Track prices, detect changes, and get email alerts. Full code with scheduler and database.
Getting Started with FineData API
Learn how to set up and make your first web scraping request with FineData API in under 5 minutes.
How to Handle CAPTCHAs When Web Scraping
Learn about different CAPTCHA types (reCAPTCHA, hCaptcha, Turnstile), how they detect bots, and strategies to handle them in your scraping pipeline.
How to Scrape Marketplace Product Data with Python
Pull titles, prices, ratings and variations from marketplace product pages in Python — hand-rolled parsing first, then structured extraction.
Python Web Scraping: Requests + BeautifulSoup vs Scraping API
Compare DIY web scraping with requests and BeautifulSoup against using a scraping API. Side-by-side code, cost analysis, and when to use each.
How to Scrape JavaScript-Heavy Websites and SPAs
Learn why traditional scrapers fail on SPAs and how to scrape React, Vue, and Angular sites using JavaScript rendering and wait strategies.
Web Scraping with Node.js: Puppeteer, Playwright, or API?
Compare Node.js web scraping approaches: Puppeteer, Playwright, and scraping APIs. Learn when to use each with practical code examples.