From the engineering desk
Notes on getting the data.
Practical tutorials, technical explainers and field notes on web scraping.
Your Scraper Died With Exit Code 137: Fix the OOM Kill
Workers die mid-batch with exit code 137 and no logged exception. Learn to confirm the OOM kill, find the memory hog, and stop buffering full page bodies.
Slow Scrape, No Timeout: MCP Progress and Cancellation
A JS-rendered page can outlast your agent's tool call timeout. Add MCP progress notifications, handle cancellation, and stop paying for abandoned scrapes.
Price the Bytes: A Storage Budget for Scraped Data
The fetch is priced; the bytes keep billing you. Model storage, gzip, egress, and retention per scraped page with numbers you can recompute.
Batch a URL List Into Partitioned Parquet Files
Turn a list of URLs into partitioned Parquet files with a staged Python pipeline: fetch, extract, validate, and load, with a run manifest you can audit.
Your Backoff Ignores Retry-After: Fix Persistent 429s
Still throttled despite exponential backoff? Learn to parse Retry-After headers, cap per-host concurrency, and stop the 429 cycle stalling your scrape pipeline.
Too Many Redirects: Fix Redirect Loops in Your Scraper
Your scraper dies with 'too many redirects' on a page that loads fine in a browser. Trace the cookie and geo causes, then cap and fix the loop.
Measure Exit IP Geolocation Accuracy With a Local Check
Run a small local audit: log the exit IP country on every scrape request, compare it to the country you asked for, and compute an accuracy score.
Throttle MCP Tool Calls: A Token Bucket for Agents
AI agents can fire MCP scraping tools in bursts that burn quota and trip rate limits. Build a token-bucket throttle that paces tool calls without stalls.
Backfill, Then Refresh: A Two-Phase Scrape Pipeline
Build a scrape pipeline that backfills once, then refreshes on a schedule. Working Python code, checkpoints, and a job queue you can run today.
Certificate Verify Failed: Fix SSL Errors in Web Scrapers
Your scraper throws SSLError: certificate verify failed. Learn the real causes — TLS inspection, stale CAs — and fix verification without turning it off.
Test Your MCP Scraping Server Before an Agent Uses It
Run your MCP scraping tools through a local test harness: verify tool discovery, argument handling, and response shape before a live agent calls them.
Amortize Maintenance Into Your Per-Page Cost
Your per-page price omits the hours spent fixing selectors and updating parsers. Learn to amortize scraper maintenance into a true per-page cost model.
Your Scraped Timestamps Say '2 Days Ago': Fix Relative Dates
Scraped pages return '2 hours ago' instead of real datetimes. Learn why relative dates break time-series data and how to normalize them at parse time.
Audit What Your Scraper Sends With a Local Echo Server
Spin up a local echo server, point your scraper at it, and log every header and cookie it sends — so you find fingerprint leaks before an anti-bot system does.
Cache MCP Tool Responses: Stop Re-Fetching the Same Page
Agents re-fetch the same URL every turn. Add a TTL cache to your MCP scraping tools, serve repeat calls instantly, and invalidate entries when data goes stale.
Too Many Open Files: Fix a Socket Leak in Your Scraper
Scrape workers crash with 'too many open files' mid-batch. Trace the socket leak, confirm it with lsof, and fix it with pooled connections and closed responses.
Build a Scrape Cost Ledger: Trace Where Dollars Went
Log a cost dimension on every fetch and build a per-URL ledger that shows where your scraping budget actually goes — renders, retries, and wasted pages.
Trip a Circuit Breaker When Your Scrape Target Degrades
Success rate tanks mid-run and retries pile up. Learn to separate target-site degradation from your own bugs, then trip a circuit breaker and fail fast.
Turn a Batch of Scraped Pages into a Queryable Dataset
Build the transform and load stages of a scraping pipeline: normalize page records, dedupe on upsert, and land them in SQLite with working code.
Your Scraped Links Don't Resolve: Fix Relative URLs
Scraped hrefs like /products/42 wreck downstream crawls and dedupe keys. Learn how to resolve relative URLs to absolute ones before you store them.
Log Your Exit IPs: Measure Proxy Rotation Locally
Spin up a local HTTP listener, point your scraper at it, and measure exit IP diversity, reuse windows, and rotation gaps before blocks happen.
Debug MCP Tool Calls: Trace What Your Agent Fetched
Wire logging and tracing into an MCP session so you can see every tool call your agent makes, inspect payloads, and pin down where live web data went wrong.
How Retries Quietly Double Your Scraping Costs
Failed attempts still cost money. Model attempts-per-success, fallback tiers, and proxy burn to compute what a delivered page truly costs.
Your 200 OK Responses Are Actually Block Pages
A 200 status code doesn't mean you got the page. Detect challenge pages and interstitials hiding inside successful responses, then fix your pipeline.
Build a Scraping Pipeline You Can Safely Restart Mid-Run
A tutorial on checkpointing batch scraping runs with run manifests and idempotent writes, so a crashed pipeline resumes exactly where it stopped.
Token Economics of LLM Extraction: A Cost Worksheet
Measure the real token cost of LLM extraction: tokens per page, per field, and per clean record, with a worksheet you can recompute on your own corpus.
Audit Field Fill Rates: Measure Extraction Quality Locally
Build a local audit script that scores scraped datasets by field fill rate, null clusters, and stale values before bad rows reach your warehouse.
Garbled Scraped Text: Fix Character Encoding Before Parsing
Your scraper returns mojibake instead of product names. Trace the failure from missing charset headers to wrong decode steps and fix it before parsing.
Throughput Decay Before the 429s: Fix Hidden Throttling
Your batch pace slows long before requests fail. Learn to tell target-site throttling from your own concurrency limits and fix it with adaptive pacing.
Break Down Scrape Latency: A Timing Harness You Can Run
Instrument a scraping pipeline from DNS to full render, log per-stage timings in Python, and compute p50/p95/p99 to find where each request spends its time.
Forecast Scrape Spend: A Budget Model You Can Recompute
Turn success rates, retry multipliers, and rendering overhead into a scraping budget you can defend, with formulas and worked numbers you can recompute.
Trim Scraped Pages Before They Flood the Agent Context
Tutorial: shrink live web pages before they reach an MCP-connected agent. Trim, chunk, and convert scraped HTML so tool output fits the context window.
Measure Your Scraper's Real Success Rate Locally
Build a small Python harness that logs every scrape attempt and computes success rate, latency percentiles, and block rates you can actually trust.
What One Successful Page Actually Costs: A Unit Model
Build a per-page cost model for scraping: retries, rendering, proxy traffic, and CAPTCHA solving, with worked numbers you can recompute for your own budget.
Re-Drive Only the Failures: Handling Partial Batch Results
A batch scrape rarely fails cleanly. Learn to reconcile partial results, classify why each URL failed, and re-drive only the misses without paying twice.
Scrape Mobile Pages Through a Carrier Exit IP
Some sites serve challenges to desktop traffic but real pages to phones. Learn when a mobile carrier exit IP, set with use_mobile, is the fix.
Your Scraper Died in a Redesign: An Extraction Escalation
A site redesign turned your working selectors into nulls overnight. Fix it with an escalation ladder: CSS rules, then schema, then prompt extraction.
Hidden JSON Endpoints vs DOM Parsing: What to Scrape
Modern sites ship data as embedded JSON. Compare parsing the DOM with tapping hidden JSON endpoints: fidelity, selector drift, and the cost of failed probes.
Fix Scraped Data That Lands in the Wrong Fields
Extracted JSON looks valid, but prices land in the wrong field. Trace the cause and fix mis-mapped data with CSS selectors, a JSON schema, or a prompt.
Sitemap Crawling vs Link Discovery: Coverage and Cost
Compare sitemap-first and link-following crawls: coverage, freshness, dead-URL waste, and how billed failures change the math for each path.
Parse at Fetch Time vs Store Raw HTML: Pipeline Trade-offs
Compare parsing scraped pages at fetch time against storing raw HTML first: reprocessing, schema drift, storage cost, and retry economics.
Batch Requests vs Single Calls: Scraping Pattern Trade-offs
Compare batching and single-request scraping patterns: throughput, retry logic, cost control, and why billing only successful requests changes the math.
On-Demand Scraping vs Prefetched Data: Serving Trade-offs
Latency, freshness, and cost per served result: when to scrape in the request path versus harvest ahead into storage, and what failed fetches cost.
Avoid CAPTCHAs vs Solve Them: Strategy Trade-offs
Two ways to deal with CAPTCHAs at scale: engineer them away with stealth or pay to solve each one. An honest comparison of cost, latency, and success rates.
CSS Selectors vs LLM Extraction: What Breaks First
Selectors are cheap and brittle; LLM extraction handles messy pages but bills per run. Compare accuracy, drift, maintenance, and the cost of failures.
Stop Cleaning HTML by Hand: Scrape Pages as Markdown
Chunking raw HTML for LLM pipelines wastes tokens and breaks parsers. See how one scrape request field returns clean, ready-to-chunk markdown instead.
Re-Scrape vs Change Detection: Choosing a Refresh Strategy
Compare scheduled full re-scrapes with change detection using ETags, sitemap lastmod, and content hashing — the bandwidth, freshness, and complexity trade-offs.
Buying Datasets vs Scraping Them Fresh: Sourcing Trade-offs
Compare buying ready-made datasets against scraping fresh data on cost, freshness, coverage, and control — and learn when a hybrid sourcing strategy wins.
Official API vs Web Scraping: Choosing a Data Source
Official API or web scraping? Compare data coverage, rate limits, pricing on failed requests, and maintenance overhead before committing to one.
Own Proxy Pool vs Managed Rotation: When to Build
Honest trade-offs between running your own proxy pool and managed rotation — uptime, block rates, and who pays for failed requests.
Client-Side vs Service-Side Retries for Scraping Failures
Compare handling failed scrape requests in your own code versus letting a scraping service retry internally — and what each approach costs per attempt.
JS Rendering vs Plain HTTP: The Real Cost of Every Scrape
Compare always-on JS rendering with plain HTTP scraping: speed, success rates, and per-request cost. Learn when rendering pays off and when it wastes budget.
Success-Based vs Metered Scraping API Billing Models
Compare success-based and pay-per-request scraping API billing: how failed requests hit your budget, and how to model real cost per successful page.
Sync vs Async Scraping: When Blocking Requests Are Enough
Compare sync and async scraping: throughput, rate limits, debugging, and cost per successful request. Learn when blocking code wins and when fan-out pays off.
Beyond GET: MCP send_http_request for Write Methods
Teach an AI agent to send POST, PUT, PATCH, and DELETE through an MCP send_http_request tool, with idempotency keys, guardrails, and retry logic.
MCP Tools for AI Agents: Auth Scopes and Failure Modes
A hands-on tutorial on designing MCP tools that let an AI agent fetch live web data: batching, token scopes, retries, and failure handling.
Scrape Localized Storefronts With a Country Exit Code
Storefronts change price, language, and stock by visitor country. Pin the scrape exit with an ISO-2 code and handle 422 when the country is unsupported.
Route Scraping Traffic Through Proxies You Already Own
When targets allowlist your IPs or you already pay for residential proxies, attach a proxy profile so scrape requests exit through your pool.
Steam 429 Error: Parsing Store Pages Responsibly with Python
Handle Steam 429 errors without retrying early: respect Retry-After, separate API and target failures, and extract Portal 2 store data with Python.
Wikipedia Scraping Returns None: Fix the Page and the Selector
Fix Wikipedia scraping that returns None: distinguish failed retrieval from Parsoid selector changes, validate extracted fields, and preserve text spacing.
Keep One Exit IP Across Multi-Step Scrape Requests
Learn how proxy_sticky keeps the same exit IP for warmup and follow-up requests, preventing session resets during multi-step scraping workflows.
Sticky Exit IPs for Warmup and Follow-Up Scrapes
Rotating proxies between a warmup hit and the next request breaks cookies and geo checks. Learn how a sticky exit IP keeps the session on the same IP.
TCP/IP Fingerprinting: OS Clues in the SYN Packet
Capture TCP/IP fingerprints from SYN fields — TTL, window, MSS, option order, DF — with a scapy sniffer on your host. See why the OS stack leaks below TLS.
Async Scraping at Scale: Jobs, Batches, Webhooks
Practical guide to FineData async scraping: submit jobs, poll with exponential backoff, verify webhook callbacks, and run batches of up to 100 URLs.
HTTP/2 Fingerprinting: Capture Client Frame Signatures
Capture HTTP/2 fingerprints from SETTINGS, WINDOW_UPDATE, PRIORITY, and pseudo-header order using a local Python listener you control.
From HTML to JSON: Schema-Driven Extraction
Turn scraped pages into validated JSON with FineData: pick extract_schema, extract_prompt, or extract_rules, tune ai_content_mode, and export CSV/XLSX.
Headless Browser Detection: What Test Pages Check
Public test pages reveal why default Playwright and Selenium fail headless checks—webdriver, WebGL, plugins, CDP—and what stealth setups change.
JA3/JA4 TLS Fingerprints Across HTTP Clients
Compare JA3 and JA4 TLS fingerprints across requests, curl, Go, Node, and browsers. Capture your own hashes with a runnable Python script.
Free No-Code Web Scraper: Extract Data Without Writing Code
How to use no-code web scrapers to extract structured data from websites. Tools, workflows, and practical limitations for non-developers.
Is There a Free Scraper API? FineData's Free Tier Explained
Discover how FineData's free tier offers real scraping power without credit card. Perfect for testing and small projects.
How to Scrape Job Postings with Dynamic Filters Using FineData API
Step-by-step guide to extract job listings from career sites with dynamic filters using FineData's API and Playwright rendering.
Web Scraper in Python: A Production Build with the FineData API
Build a production-grade Python web scraper that handles JavaScript rendering, anti-bot systems, and structured extraction using FineData's API, with working code examples.
How to Scrape Dynamic Product Feeds from Storefronts
Extract real-time product data from hosted storefronts that render their catalog in JavaScript — browser rendering, sticky sessions, batch jobs.
Anti-Bot Detection: How Cloudflare, DataDome, and PerimeterX Work
How modern anti-bot systems detect scrapers in 2026: IP reputation, TLS fingerprinting, JS challenges, behavioral analysis, and device fingerprinting explained.
Building a Price Monitoring Tool: Step-by-Step Guide
Build a complete price monitoring tool with Python. Track prices, detect changes, and get email alerts. Full code with scheduler and database.
Building ETL Pipelines with Web Scraping APIs
Learn how to build production-ready ETL pipelines using web scraping APIs. Covers extraction, transformation, loading, scheduling, and monitoring.
Competitive Intelligence: How to Monitor Competitors at Scale
A strategic guide to building competitive intelligence systems that monitor competitor pricing, products, content, hiring, and more using web scraping.
E-commerce Price Intelligence: Complete Strategy Guide
A strategic guide to building e-commerce price intelligence systems using web scraping for competitive pricing, dynamic pricing, and market monitoring.
The Future of Web Scraping: AI, LLMs, and Structured Extraction
Explore how AI and large language models are transforming web scraping with natural language queries, intelligent extraction, and the MCP protocol.
Getting Started with FineData API
Learn how to set up and make your first web scraping request with FineData API in under 5 minutes.
How to Handle CAPTCHAs When Web Scraping
Learn about different CAPTCHA types (reCAPTCHA, hCaptcha, Turnstile), how they detect bots, and strategies to handle them in your scraping pipeline.
Scraping Job Boards for Market Intelligence: A Complete Guide
Learn how to scrape public job boards for hiring trends, salary ranges, and market intelligence, with practical code examples and a legal checklist.
Lead Generation and B2B Data Enrichment: From Web Data to CRM
How sales teams turn public web data into a scored, enriched lead pipeline — sourcing, extraction, technology-stack detection, deduplication, CRM sync, and the GDPR/CCPA rules that apply to B2B contacts.
Automating Market Research with Web Scraping APIs
Learn how to automate market research using web scraping APIs — from building research pipelines to sentiment analysis and competitive mapping.
How to Scrape Marketplace Product Data with Python
Pull titles, prices, ratings and variations from marketplace product pages in Python — hand-rolled parsing first, then structured extraction.
MCP Protocol: How to Connect AI Agents to Web Data
Guide to the Model Context Protocol (MCP) for connecting AI agents to live web data. Set up FineData's MCP server with Cursor IDE and Claude Desktop.
Proxy Rotation Strategies for Large-Scale Web Scraping
Technical guide to proxy rotation for web scraping: datacenter vs residential vs mobile proxies, rotation strategies, IP ban detection, and cost optimization.
Python Web Scraping: Requests + BeautifulSoup vs Scraping API
Compare DIY web scraping with requests and BeautifulSoup against using a scraping API. Side-by-side code, cost analysis, and when to use each.
Building a Real Estate Data Pipeline with Web Scraping
Learn how to build an automated real estate data pipeline using web scraping to track listings, prices, and market trends at scale.
Scaling Web Scraping from 1K to 10M Pages per Day
Architecture guide for scaling web scraping from thousands to millions of pages per day. Covers async patterns, queues, rate limiting, distributed systems, and cost optimization.
How to Scrape JavaScript-Heavy Websites and SPAs
Learn why traditional scrapers fail on SPAs and how to scrape React, Vue, and Angular sites using JavaScript rendering and wait strategies.
Web Scraping API vs DIY: Total Cost of Ownership Analysis
Detailed cost comparison of building web scraping infrastructure in-house vs using a scraping API. Includes developer time, proxies, CAPTCHAs, and maintenance.
Selenium vs Puppeteer vs Playwright vs Scraping API: Complete Comparison
Head-to-head comparison of Selenium, Puppeteer, Playwright, and scraping APIs for web scraping. Architecture, performance, anti-bot handling, and scaling.
SERP Tracking for SEO: Tools, Methods, and Best Practices
Learn how to build a SERP tracking system to monitor search rankings, track SERP features, and improve your SEO strategy with web scraping.
Social Media Data Collection: Ethics, Techniques, and Best Practices
A responsible guide to collecting social media data for brand monitoring, sentiment analysis, and research while respecting ethical boundaries.
TLS Fingerprinting Explained: How Anti-Bot Systems Detect Scrapers
TLS fingerprinting explained: how JA3/JA4 hashes let Cloudflare, Akamai, and other anti-bot systems detect scrapers from the TLS client hello alone.
Build Your Own Travel Price Comparison Tool
Learn how to build a travel price comparison tool that monitors flight and hotel rates across booking platforms using a web scraping API.
Web Scraper for Academic Research: Tools and Ethics
How to run a web scraper for academic research: tool selection, IRB documentation, reproducibility, and the ethics reviewers actually check before publication.
Web Scraping Legal Guide: GDPR, CCPA, and robots.txt
Legal guide to web scraping in 2026: court cases, GDPR, CCPA, robots.txt, Terms of Service, and best practices for compliant and ethical data collection.
Web Scraping with Node.js: Puppeteer, Playwright, or API?
Compare Node.js web scraping approaches: Puppeteer, Playwright, and scraping APIs. Learn when to use each with practical code examples.
Web Scraping for Startups: Getting Data-Driven from Day One
How startups can use web scraping for market validation, competitor analysis, lead generation, and building data-driven products on a budget.
No posts in this category yet
Check back soon or browse all posts.