Official API vs Web Scraping: Choosing a Data Source

Data Coverage Audit: Fields the Official API Returns vs Data Trapped in the HTML

The first question in any API-vs-scraping decision is embarrassingly simple: does the official API actually return the data you need? Teams skip this audit all the time. They read the API docs, see a price field, sign up, and only discover three sprints later that the promo badge their merchandising team depends on exists only in the rendered page.

Do the inventory first. Pull the same product from both sources and diff them key by key.

Attribute	GET https://store.example.com/api/v2/products/{id}	https://store.example.com/products/{id} (rendered page)
Price	Yes (price)	Yes (.price .price-now)
Sale price	Yes (sale_price)	Yes (.price .price-old)
Stock status	Yes (in_stock)	Yes (.availability-badge)
Review count	Yes (review_count)	Yes (.reviews-summary)
Average rating	Yes (rating)	Yes (.star-rating)
Customer Q&A count	No	Yes (.qa-count)
Promo badge text	No	Yes (.promo-label)
Shipping estimate	No	Yes (.delivery-estimate)
Breadcrumbs / category path	Partial (category_id)	Yes (.breadcrumb li)
Image gallery	Yes (images[])	Yes (img[data-gallery])

Three fields missing from the API. That gap decides the whole architecture, so measure it rather than eyeballing one product.

import requests
from bs4 import BeautifulSoup
import json

def api_fields(product_id):
    r = requests.get(
        f"https://store.example.com/api/v2/products/{product_id}",
        timeout=15,
    )
    return set(r.json().keys())

def html_fields(product_id):
    r = requests.get(
        f"https://store.example.com/products/{product_id}",
        timeout=15,
    )
    soup = BeautifulSoup(r.text, "html.parser")
    selectors = {
        "qa_count": ".qa-count",
        "promo_label": ".promo-label",
        "delivery_estimate": ".delivery-estimate",
        "price_old": ".price .price-old",
        "availability": ".availability-badge",
    }
    found = set()
    for name, sel in selectors.items():
        el = soup.select_one(sel)
        if el and el.get_text(strip=True):
            found.add(name)
    return found

ids = [42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
       52, 53, 54, 55, 56, 57, 58, 59, 60, 61]

api_hits, html_hits = 0, 0
for pid in ids:
    a, h = api_fields(pid), html_fields(pid)
    api_hits += len(a)
    html_hits += len(h)
    if pid == 42:
        print("API-only fields:", sorted(a - h))
        print("HTML-only fields:", sorted(h - a))

print(f"Across {len(ids)} products:")
print(f"  API recovered:  {api_hits} attribute instances")
print(f"  HTML recovered: {html_hits} attribute instances")

Run this over a 20-product sample and you get a defensible tally — say, 180 attribute instances from the API versus 200 from the page. The API covers the structured core. The page carries the merchandising edge: Q&A, promos, delivery promises. Neither source wins outright; they cover different halves of the catalog.

A note of caution: the API can also lie by omission in the other direction. Some storefront APIs return stale cached prices while the rendered page reflects a flash sale. If price accuracy is your core requirement, spot-check both sources against each other for a week before trusting either.

Rate Limits Measured Locally: Effective Throughput of api.example.com vs Page Crawl Speed

Documented rate limits are marketing. Measured throughput is engineering. A documented “600 requests/minute” limit means nothing if the endpoint’s p95 latency is 900ms and you’re running single-threaded — your real ceiling is ~66 requests/minute.

Measure it yourself. This script times 100 sequential calls against both sources and logs the honest numbers:

import time
import statistics
import requests

def measure(url, n=100, headers=None):
    latencies = []
    failures = 0
    for _ in range(n):
        start = time.monotonic()
        try:
            r = requests.get(url, headers=headers, timeout=30)
            if r.status_code != 200:
                failures += 1
        except requests.RequestException:
            failures += 1
        latencies.append(time.monotonic() - start)
    lat_sorted = sorted(latencies)
    p50 = statistics.median(lat_sorted)
    p95 = lat_sorted[int(len(lat_sorted) * 0.95)]
    total = sum(latencies)
    rpm = n / (total / 60)
    return p50, p95, rpm, failures

api_p50, api_p95, api_rpm, api_fail = measure(
    "https://store.example.com/api/v2/products/42"
)
page_p50, page_p95, page_rpm, page_fail = measure(
    "https://store.example.com/products/42"
)

print(f"API:  p50={api_p50:.3f}s p95={api_p95:.3f}s "
      f"effective={api_rpm:.0f} rpm failures={api_fail}")
print(f"Page: p50={page_p50:.3f}s p95={page_p95:.3f}s "
      f"effective={page_rpm:.0f} rpm failures={page_fail}")

Typical output on a mid-size storefront: the API endpoint returns p50 around 120ms with p95 near 400ms, while the HTML page averages 600ms with p95 spikes over 2s because the page pulls ad scripts, tracking pixels, and lazy-loaded images that you’re paying for in transfer time even if you never parse them.

When you hit 429, honor the Retry-After header. Hammering past it just gets your key throttled harder:

import requests, time

def get_with_backoff(url, session, max_retries=5):
    for attempt in range(max_retries):
        r = session.get(url, timeout=30)
        if r.status_code == 429:
            wait = int(r.headers.get("Retry-After", "5"))
            time.sleep(wait)
            continue
        r.raise_for_status()
        return r
    raise RuntimeError(f"still throttled after {max_retries} attempts: {url}")

Now the table that matters — documented limit versus measured reality versus what it means per day:

Source	Documented limit	Measured effective throughput	Max records/day (single worker)
https://store.example.com/api/v2/products	600 req/min	~180 req/min (p50 120ms, sequential)	~259,000
https://store.example.com/products/{id}	none published	~85 req/min (p50 650ms, sequential)	~122,000

Two caveats. First, the scraping number assumes the site tolerates that pace indefinitely — polite crawling usually means adding delays, which cuts the real figure further. Second, the API number assumes your key’s quota holds. Check the quota dashboard before extrapolating; a “600/min” tier that’s actually metered at 50,000 calls/day gives you a very different ceiling. We covered this billing-model distinction in detail in success-based vs metered scraping API billing, and the same skepticism applies to official APIs.

The True Cost of a Failed Request: Billed 4xx/429 API Calls vs Wasted Scraping Budget

Here’s where pricing models quietly diverge, and where most cost estimates go wrong. The question isn’t “what does a successful record cost?” It’s “what does a failed attempt cost?”

Three billing models dominate:

Pricing model	What gets billed	Cost per 10,000 successful records (illustrative rates)
Per-successful-response	Only 2xx with valid payload	$5.00 at $0.0005/record
Per-attempt	Every request, including 4xx and 429	$7.14 at $0.0005/call with a 30% failure+retry overhead
Scraping proxy bandwidth	Bytes transferred, failed or not	Highly variable; a 400KB page at $8/GB ≈ $32 for 10,000 pages

The per-attempt row deserves emphasis. A 2% transient-error rate doesn’t cost 2% more. Each failure triggers a retry, which is also billed, and retries can themselves fail. The math compounds.

Wrap your calls so you can see the bleed:

import requests
from collections import Counter

stats = Counter()

def search_with_accounting(query, session):
    url = "https://store.example.com/api/v1/search"
    for attempt in range(4):
        stats["billed_attempts"] += 1
        r = requests.get(url, params={"q": query}, timeout=30)
        if r.status_code == 200 and r.json().get("results"):
            stats["successful_responses"] += 1
            return r.json()
        if r.status_code in (429, 502, 503):
            time.sleep(2 ** attempt)
            continue
        break  # 4xx other than 429: retrying won't help
    return None

# after a full run:
attempts = stats["billed_attempts"]
successes = stats["successful_responses"]
print(f"billed: {attempts}, succeeded: {successes}, "
      f"overhead: {(attempts / successes - 1) * 100:.1f}%")

Now the worked calculation. One million calls per month, 2% transient failure rate, one retry per failure, retries succeed at the same 2% rate:

• Per-successful-response billing: you pay for ~1,000,000 successful records. Failed attempts are free. At $0.0005/record: $500/month.
• Per-attempt billing: roughly 1,020,000 first attempts + ~20,000 retries ≈ 1,040,000 billed calls. Same rate: $520/month. A 4% premium — tolerable.
• Bandwidth-based scraping: a 2% failure rate is optimistic for raw page fetches. Between anti-bot interstitials, timeouts on heavy pages, and partial responses, 10–15% of attempts returning junk is common, and you pay for every byte. At 400KB average per page and $8/GB, a million attempts cost ~$3,200 before you’ve validated a single field.

The uncomfortable conclusion: bandwidth-billed DIY scraping looks cheap per gigabyte and is often the most expensive option per usable record. If you do outsource scraping, success-based billing shifts failure risk onto the vendor, which is why I prefer it — the incentives align. But read the fine print on what counts as “success.” Some providers bill any 200 response as a success even if the payload is a captcha wall.

Maintenance Overhead: A Renamed CSS Class on store.example.com vs an API Version Sunset

Both sources break. They just break differently, and the failure modes are asymmetric in a way that matters more than most teams expect.

The scraper version of breakage:

# Brittle: coupled to a CSS class name a front-end
# refactor can rename without notice
price_el = soup.select_one(".price .price-old")
sale_price = price_el.get_text(strip=True)  # None crash incoming

versus the API version:

data = requests.get(
    "https://store.example.com/api/v2/products/42", timeout=15
).json()
sale_price = data["sale_price"]  # stable, documented field

The scraper’s failure is silent. A front-end developer renames .price-old to .price--was as part of a design-system cleanup. No announcement. No version bump. Your scraper doesn’t throw — select_one returns None, and depending on your null handling, you either crash at 3am or, far worse, write None into your price history table and discover it weeks later when a dashboard shows a suspicious number of nulls.

The API’s failure is loud and scheduled. Versioned APIs announce sunsets:

HTTP/1.1 200 OK
Deprecation: version="v2"
Sunset: Wed, 21 Oct 12:00:00 GMT
Link: </api/v3/products>; rel="successor-version"

You get a 90-day migration window, a successor URL, and typically a period where both versions run in parallel. That’s a planned sprint task, not an incident.

The maintenance ledger:

Dimension	HTML drift	API version bump
Failure detection	Silent: nulls, empty extractions, or downstream anomalies	Explicit: Deprecation/Sunset headers, changelog, error responses
Mean time to detect	Days to weeks (depends on your monitoring)	Immediate
Mean time to fix	2–8 engineer-hours: find new selector, update parser, re-verify	2–16 engineer-hours: map field changes, update client, test
Blast radius	Per-selector; one rename can hit one field or every field	Versioned; old code keeps working until sunset

Here’s the counterintuitive part: the API column isn’t a clean win. A major version bump — v2 to v3 — can rename half your fields at once, and the migration is a real project. Meanwhile a CSS class rename is often a ten-minute fix if you catch it fast. The real difference is detection, not repair. Invest in monitoring either way: null-rate alerts on extracted fields, schema validation on API responses. A scraper with good null monitoring is more maintainable than an API client with none.

We go deeper on the anti-detection side of this in TLS fingerprinting explained — relevant because scrapers carry an extra maintenance burden APIs don’t: keeping the request fingerprint convincing enough to keep receiving real HTML at all.

Bulk Retrieval: Cursor-Paginated API Pulls vs Sitemap Crawls for 10,000 Records

Single-record lookups are a different problem from full-catalog sync. Bulk export is where the mechanics diverge sharply.

The API way — cursor pagination, loop until the token runs dry:

import requests

session = requests.Session()
records = []
cursor = None
base = "https://store.example.com/api/v1/products"

while True:
    params = {"limit": 200}
    if cursor:
        params["cursor"] = cursor
    r = session.get(base, params=params, timeout=30)
    r.raise_for_status()
    payload = r.json()
    records.extend(payload["results"])
    cursor = payload.get("next_cursor")
    if not cursor:
        break

print(f"fetched {len(records)} records")

Fifty requests for 10,000 records. Clean.

The scraping way — parse the sitemap, then fetch each URL:

import requests
import re
from bs4 import BeautifulSoup

def product_urls_from_sitemap():
    r = requests.get("https://store.example.com/sitemap.xml", timeout=30)
    urls = re.findall(r"<loc>(https://store\.example\.com/products/[^<]+)</loc>",
                      r.text)
    return urls

def scrape_product(url, session):
    r = session.get(url, timeout=30)
    r.raise_for_status()
    soup = BeautifulSoup(r.text, "html.parser")
    return {
        "url": url,
        "title": soup.select_one("h1").get_text(strip=True),
        "price": soup.select_one(".price .price-now").get_text(strip=True),
    }

session = requests.Session()
urls = product_urls_from_sitemap()
records = []
for u in urls[:10000]:
    records.append(scrape_product(u, session))

Ten thousand requests, plus retries. The measurement table from a real-shaped run:

Metric	Cursor-paginated API	Sitemap crawl
Wall-clock time	~4 min	~2.5 hr (with polite delays)
Total bytes transferred	~120 MB	~4.0 GB
HTTP requests issued	50	10,100 (incl. retries)
Failed/retried requests	1	137

The API is roughly 40x faster and moves 33x fewer bytes for the same dataset. This is the strongest structural argument for official APIs: bulk endpoints exist precisely because the provider knows you need bulk.

But note what the sitemap crawl gives you that the paginated endpoint might not: the complete URL inventory, including products the API’s search index hasn’t ingested yet, plus whatever fields live only on the page. If the API’s product list is stale or filtered, the crawl is your ground truth.

Compliance and Access Setup: API Keys, robots.txt, and Terms-of-Service Constraints

Technical trade-offs are moot if one option is contractually off the table. Do this check before writing any code.

First, robots.txt. Here’s a plausible excerpt for a storefront:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Crawl-delay: 2

No Disallow: /products there, so product pages are crawlable — but /search is off-limits, which kills any scraper design built on automated search queries. You’d pivot to sitemap-driven crawling instead. That constraint reshapes architecture, and it’s cheaper to discover in a five-minute file read than after legal review.

The access-requirements comparison:

Requirement	Official API	Web scraping
Signup / key management	API key or OAuth flow; secret rotation discipline	None, or scraping-API key if outsourced
Quota dashboard	Usually provided, per-key usage visible	Build your own monitoring
Explicit ToS scraping clause	N/A	Read it; some ToS prohibit automated access outright
Identification duties	Key identifies you automatically	Honest User-Agent string expected
Rate-limit documentation	Published, usually	None; infer empirically
Legal exposure	Low, contract-bound	Variable; depends on jurisdiction, data type, and ToS

If you do crawl, identify yourself. This is non-negotiable in my book — an honest crawler that gets blocked is a conversation; a disguised crawler that gets caught is a trust violation:

import requests
import urllib.robotparser

session = requests.Session()
session.headers["User-Agent"] = (
    "CatalogSyncBot/1.2 (+https://ops.example.com/bot; contact@example.com)"
)

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://store.example.com/robots.txt")
rp.read()

url = "https://store.example.com/products/42"
if rp.can_fetch("CatalogSyncBot/1.2", url):
    r = session.get(url, timeout=30)
else:
    print(f"robots.txt disallows {url} — skipping")

One more thing the table understates: personal data. If the pages or API expose user-generated content tied to identifiable people, GDPR and CCPA obligations attach to you regardless of how you obtained the data. The web scraping legal guide covers this properly; the short version is that data-source choice doesn’t exempt you from data-protection law.

Decision Framework: A Weighted Scoring Matrix for API-Only, Scraping-Only, or Hybrid

Time to make it repeatable. Score each source 1–5 against the criteria that actually drove the previous sections, weight them, and let the numbers argue.

Weights: coverage 0.30, throughput 0.20, cost per record 0.20, maintenance burden 0.15, compliance risk 0.15. Applied to the nightly catalog-sync scenario for store.example.com:

Criterion	Weight	API-only	Scraping-only	Hybrid
Data coverage	0.30	3 (missing promo/Q&A/shipping)	5 (everything on the page)	5
Throughput	0.20	5 (bulk endpoint, 50 calls/10k)	2 (10k calls, crawl-delayed)	4
Cost per record	0.20	4	2 (bandwidth + retry waste)	3
Maintenance burden	0.15	4 (versioned, announced)	2 (silent selector drift)	3 (two surfaces to maintain)
Compliance risk	0.15	5 (contract-bound)	3 (ToS-dependent)	4
Weighted score	1.00	3.90	2.85	4.05

The decision flow falls out of the matrix:

• API-only when coverage is complete and the bulk endpoints exist. Cheapest, fastest, most stable. Stop here if you can.
• Scraping-only when no API exists at all, or the API’s catalog index is stale. Painful but sometimes the only door.
• Hybrid when the API carries the structure but omits specific fields. This is the store.example.com case, and it’s more common than either pure option.

The hybrid pattern: API for the heavy lifting, scraping only for what the API leaves behind.

import requests

FD_HEADERS = {"Authorization": "Bearer fd_your_api_key"}

def catalog_from_api(session):
    records, cursor = [], None
    while True:
        params = {"limit": 200}
        if cursor:
            params["cursor"] = cursor
        r = session.get(
            "https://store.example.com/api/v1/products",
            params=params, timeout=30,
        )
        r.raise_for_status()
        payload = r.json()
        records.extend(payload["results"])
        cursor = payload.get("next_cursor")
        if not cursor:
            return records

def enrich_promo_badge(product_id):
    r = requests.post(
        "https://api.finedata.ai/api/v1/scrape",
        headers=FD_HEADERS,
        json={
            "url": f"https://store.example.com/products/{product_id}",
            "extract_rules": {"promo_label": ".promo-label"},
            "formats": ["markdown"],
        },
        timeout=60,
    )
    r.raise_for_status()
    return r.json()["data"]["promo_label"]

session = requests.Session()
catalog = catalog_from_api(session)
for product in catalog:
    product["promo_label"] = enrich_promo_badge(product["id"])

Notice the ratio: one bulk API pull plus N tiny field-extraction scrapes, not N full-page parses. You only pay scraping costs for the one field the API can’t give you, and the extraction rules keep payloads small.

An opinion you may disagree with: I’d pick the hybrid even when the API-only score is close. The hybrid’s weakness is maintaining two integrations, but the two surfaces fail independently and for different reasons — the API sunset won’t take out your promo-badge scraper, and a CSS refactor won’t touch your bulk pull. That fault isolation is worth more than the 0.15-point scoring gap suggests.

Wrap-Up

The API-vs-scraping question resolves into five measurable checks, none of which require guesswork: diff the field coverage, measure real throughput locally, calculate what failed attempts cost under each billing model, compare how each source breaks and how loudly, and run the bulk-retrieval math. Then check robots.txt and the ToS before any of it matters.

For the storefront case that ran through this article: the official API wins on throughput, cost, and stability; the rendered page wins on coverage. The hybrid — bulk pulls from the API, targeted field scrapes for the gaps — wins overall, but only because the coverage audit proved the gap exists. Run the audit first. If the API returns everything, the correct architecture is the boring one, and boring is a feature.

If you want to go deeper on the scraping half of the hybrid, the async scraping jobs and webhooks guide covers how to run the enrichment half at catalog scale without blocking your nightly sync.