Success-Based vs Metered Scraping API Billing Models

Introduction

Scraping API pricing looks simple until the first invoice disagrees with your spreadsheet. Two providers can quote the “same” per-page price and cost you wildly different amounts, because one charges for attempts and the other charges for outcomes. The gap between those two models is entirely a function of your failure rate — a number most teams never measure before signing a contract.

This post compares success-based billing (pay only for usable responses) against metered billing (pay per request, regardless of outcome), shows how to compute your real cost per successful page under each, and gives you the break-even math to pick correctly. If you’re still deciding whether a scraping API makes sense versus running your own fleet, the total cost of ownership analysis covers the infrastructure side; here we’re focused purely on how the billing model interacts with target fragility.

How Success-Based Billing Works: Paying Only for HTTP 200s

Under success-based pricing, a request is billable only when it returns a usable response. You submit the job, the provider routes it through their proxy and anti-bot stack, and money changes hands only if the target returns real content. Blocks, timeouts, and origin errors are the provider’s problem, not yours.

import requests

API = "https://api.finedata.ai/api/v1/scrape"
headers = {"Authorization": "Bearer fd_your_api_key"}

# timeout=130: generous window so the provider's internal retries finish before we give up
r = requests.post(API, headers=headers, json={
    "url": "https://example.com",
    "formats": ["markdown"],
}, timeout=130)

# Under success-based billing:
#   200 with content        -> billable
#   blocked / timeout /
#   empty body / 5xx        -> not billable

What counts as “success” is where the model gets slippery. The generous interpretation — and the one you should demand in writing — is a 200 response with non-empty, non-block content. Some providers count any HTTP 200, including a 200 that’s actually an anti-bot interstitial page. That distinction is worth real money, and we’ll come back to it in the fine print section.

For the standard status-code interpretation:

Status	Meaning	Billable under success-based?
200	Full page returned	Yes
404	Page doesn’t exist	No
429	Rate limited	No
503	Origin unavailable / challenged	No

The economics are straightforward: the provider absorbs failure risk, so they price each success at a premium — typically 3-5x the metered per-request rate. You’re buying insurance against fragile targets, and the premium reflects that.

How Metered (Pay-Per-Request) Billing Works: Every Attempt Counts

Metered pricing charges per API call. Full stop. The request can come back with a perfect page, a 403 block, or nothing at all — the charge lands either way. This is the model most people intuitively expect, and it’s the model that quietly punishes sloppy retry logic.

Here’s the trap in code form:

import requests, time

target = "https://store.example.com/search?q=laptops"

for attempt in range(3):
    try:
        r = requests.get(target, timeout=30)
        if r.status_code == 200 and "laptop" in r.text.lower():
            break
    except requests.RequestException:
        pass
    time.sleep(2 ** attempt)

# Three attempts left the building. Under metered pricing,
# all three are charged -- even if attempts 1 and 2 returned
# anti-bot blocks with zero usable content.

Nothing in that loop looks wrong. Exponential backoff, a sanity check on the response body, a clean break on success. But if the target is having a bad day and blocks two of three attempts, you paid triple for that page.

The flat-cost property is the model’s main selling point: 10,000 requests at $0.001 each = $10, regardless of how many succeeded. That predictability per request becomes unpredictability per successful page the moment your failure rate moves. A $10 invoice for 10,000 requests is fine if 9,800 returned content. It’s terrible if 4,000 did.

The Hidden Cost of Failures: Why Metered Pricing Punishes Fragile Targets

Failure costs are invisible in metered pricing because the invoice never itemizes them. You see “142,000 requests” and a dollar figure. You don’t see that 38,000 of those requests returned nothing you could use. Here’s what a bad hour against a protected target actually looks like in the logs:

14:22:31 GET /search?q=laptops   -> 403 (blocked)
14:22:33 GET /search?q=laptops   -> 429 (rate limited)
14:22:36 GET /search?q=laptops   -> 200 (152 KB)
14:22:37 GET /search?q=monitors  -> 403 (blocked)
14:22:39 GET /search?q=monitors  -> 200 (148 KB)

Five log lines, two successes, two blocks, one rate limit. Under metered pricing that’s five charges for two usable pages. Under success-based pricing it’s two charges for two usable pages. Now scale that ratio across a crawl.

Same 50,000-page job, run under both models. Metered at $0.001 per request, success-based at $0.004 per success, one attempt per page:

Failure rate	Metered spend	Metered cost per success	Success-based spend	Success-based cost per success
0%	$50	$0.00100	$200	$0.004
5%	$50	$0.00105	$190	$0.004
20%	$50	$0.00125	$160	$0.004
40%	$50	$0.00167	$120	$0.004

Two things jump out. First, metered total spend is flat — failures don’t touch the invoice, they only trash your effective cost per page, which climbs 67% as the failure rate goes from 0% to 40%. Second, and this is the part people miss: success-based spend falls as failures rise. The provider eats the loss. That’s exactly why the per-success rate is 4x the per-request rate in this example — the premium is priced to cover the failures of their average customer, not yours.

Which brings up the uncomfortable question: if your failure rate is lower than the average customer’s, you’re subsidizing someone else’s broken pipeline. If it’s higher, you’re the one being subsidized. Know which one you are.

Modeling Real Cost per Successful Page: The Effective-CPA Formula

The only number that matters for comparing billing models is effective cost per successful page. Not the quoted rate. Not the invoice total. What you paid divided by what you actually got.

# Metered model:
effective_cost_per_success = (total_requests × price_per_request) / successful_responses

# Success-based model:
effective_cost_per_success = per_success_rate   # constant by construction

A worked example with realistic numbers. You run 100,000 requests against a mid-fragility target. 82% come back with usable content. Your metered rate is $0.0015 per request.

• Total spend: 100,000 × $0.0015 = $150
• Successful pages: 82,000
• Effective cost per success: 150 / 82,000 = $0.00183 per page

Now compare against a success-based plan at $0.004 per success. The same 82,000 pages cost $328. The metered plan wins by a factor of more than two — at this failure rate. Push the failure rate to 40% and the metered effective cost rises to $0.0025 per page. Still cheaper, but the margin is eroding, and retries (covered below) will eat into it faster than the base failure rate suggests.

The formula also tells you what to negotiate. If a vendor quotes you a per-request rate, your first question should be “what’s your observed success rate against targets like mine?” If they can’t answer, they either don’t measure it or the answer is bad. Either response is informative.

Measuring Your Own Failure Rate Before Choosing a Model

Don’t pick a billing model off vendor marketing. Run a pilot against your actual targets, log every outcome, and compute the failure rate yourself. This takes an afternoon and it’s the highest-ROI afternoon you’ll spend on scraping economics.

import csv, time, requests

TARGETS = [
    "http://example.com/category/tools",
    "http://example.com/category/power-drills",
]
max_attempts = 3

with open("scrape_log.csv", "a", newline="") as out:
    writer = csv.writer(out)
    ok = fail = 0

    for url in TARGETS:
        attempts, status = 0, None
        while attempts < max_attempts:
            attempts += 1
            try:
                r = requests.get(url, timeout=30)
                status = r.status_code
                if status == 200:
                    break
            except requests.RequestException as e:
                status = type(e).__name__
            if attempts < max_attempts - 1:
                time.sleep(2 ** attempts)
        if status == 200:
            ok += 1
        else:
            fail += 1
        writer.writerow([time.strftime("%H:%M:%S"), url, status, attempts])
        out.flush()

print(f"success={ok} fail={fail} rate={ok / (ok + fail):.3f}")

The resulting log gives you exactly the inputs the effective-CPA formula needs:

timestamp,url,status,attempts
14:22:31,http://example.com/category/tools,200,1
14:22:36,http://example.com/category/power-drills,403,3
14:22:41,http://example.com/category/sanders,200,2

Two practical notes. First, run the pilot at the same concurrency you plan to use in production — failure rates on protected targets are concurrency-sensitive, and a serial pilot will flatter your numbers. Second, if you’re already routing through a scraping API, reconcile your local log against the provider’s counters. FineData exposes GET /api/v1/usage for balance and consumption plus GET /api/v1/user/stats for request history over a date range, so you can cross-check what you logged locally against what was actually billed. If those two numbers diverge, you’ve found either a bug in your logging or a billing definition you didn’t read carefully enough. Both are worth finding before the crawl scales.

Retry Strategies Change the Math: Why Retries Are Free Under One Model and Expensive Under the Other

Retries are where the two models genuinely diverge, because retries multiply attempts but not successes. A retry policy that looks conservative on paper can double a metered invoice against a fragile target.

retry_policy:
  max_retries: 3
  backoff: exponential
  base_delay_seconds: 2
  retry_on: [429, 500, 502, 503, timeout]
  # Metered model: every retry is a fresh charge. A dead target with
  #   max_retries=3 means up to 4x the billable attempts for that page.
  # Success-based model: failed retries cost nothing; you pay once,
  #   on the attempt that finally succeeds.

Here’s a 10,000-page job with a 20% per-attempt failure rate, showing total billable units under each model:

max_retries	Metered billable attempts	Success-based billable successes
0	10,000	8,000
1	12,000	9,600
2	12,400	9,920
3	12,480	9,984

Read the metered column carefully: retries inflate it by up to 25% even at a modest 20% failure rate, and the growth compounds badly at higher failure rates. At 50% per-attempt failure, max_retries: 3 produces 18,750 billable attempts for 10,000 pages — nearly double.

The success-based column moves too, and that’s the honest counterpoint. More retries means more pages eventually succeed, and every success is billed. But that column is capped at 10,000 — it cannot exceed the page count — while the metered column has no ceiling. There’s also a subtler effect: success-based providers know retries are free for you, so they tend to cap retry counts or throttle aggressive customers to protect their own margin. Read your contract for retry limits; they’re often the real constraint, not the price.

If you’re on a metered plan, tune retries against the retry section of your target’s behavior and use rotation to prevent failures in the first place rather than retrying your way out of them — the strategies in proxy rotation strategies for large-scale scraping prevent the blocks that make retries expensive.

When Each Model Wins: Decision Criteria and Break-Even Points

Time for the actual math that settles the choice. With metered at $0.001 per request and success-based at $0.004 per success, your metered cost per successful page is:

0.001 / (1 - failure_rate)

Set that equal to $0.004 and solve:

0.001 / (1 - f) = 0.004
1 - f = 0.25
f = 0.75

The break-even failure rate is 75%. Below that, metered is cheaper. Above it — a failure rate where three out of four requests die — success-based wins. Think about what that means in practice: a 75% failure rate isn’t a pricing problem, it’s an architecture problem. You should not be scraping that target with either billing model until you’ve fixed your approach, because at that failure rate your pipeline is mostly generating heat.

This is the opinion part, and you may disagree: success-based pricing is usually a bad buy on price alone. The per-success premium is priced to cover the average customer’s failure rate. If you’ve measured your failure rate and it’s under 30% — which it will be for any target where you’ve done the anti-detection homework — you’re paying an insurance premium for risk you don’t have. The legitimate reasons to choose success-based are predictability (finance gets a flat cost per page) and pilots against unknown targets where you genuinely don’t know the failure rate yet.

Scenario	Recommended model
Stable, API-like target, >95% success	Metered
Protected target like store.example.com behind aggressive anti-bot	Metered with capped retries, or success-based if unmeasured
New target, unknown fragility	Success-based for the pilot, then renegotiate
Fixed budget, one-off crawl	Success-based (predictability)
High volume, tuned pipeline, measured <30% failure	Metered, always

Reading the Fine Print: Success Definitions, Minimums, and Overage Traps

The billing model is only half the contract. The other half is the definitions, and that’s where the money leaks. Three clauses deserve line-by-line scrutiny:

Clause	Hypothetical success-based plan	Hypothetical metered plan
”Success” definition	HTTP 200 with non-empty body — CAPTCHA interstitials returning 200 count as success	N/A; every request billed, including 200s that are block pages
Monthly minimum	$500/mo; unused success credits expire monthly	$100/mo
Overage rate	$0.005 per success beyond plan (25% above in-plan rate)	$0.0012 per request beyond plan (20% above in-plan rate)

The success definition is the biggest trap. A provider who counts “HTTP 200” as success can serve you a CAPTCHA page with a 200 status and bill you for it. The definition you want is “200 with content matching the requested format” — and if the vendor can’t state their definition precisely, assume the worst.

Monthly minimums interact with failure rates in a nasty way on success-based plans: if your crawl succeeds less than expected, you burn fewer credits, and expiring credits mean you paid for successes that never happened. Overage rates matter more the more successful you are — a cheap in-plan rate with a punitive overage rate is a trap for growing pipelines.

When you evaluate plans, ask for the pricing as structured data so you can model it, not as a sales page. Something like:

{
  "plan": "scraper-pro",
  "billing_model": "success_based",
  "per_success_rate": 0.004,
  "per_request_rate": null,
  "included_successes": 150000,
  "overage_per_success": 0.005,
  "success_definition": "http_200_non_empty_body",
  "monthly_minimum_usd": 500,
  "unused_credits_rollover": false
}

If a vendor can’t produce this, build it yourself from the contract and run your measured failure rate through both models before signing. FineData publishes its plans via GET /api/v1/plans, which at least lets you pull the numbers programmatically instead of copying them off a pricing page — but the definitions still need a human read.

Wrap-Up

The billing model decision reduces to one measured number: your failure rate. Metered pricing is cheaper for almost every pipeline that has done its anti-detection homework, because the success-based premium is an insurance product priced for the average customer, not for you. Success-based pricing earns its premium in exactly three situations: pilots against unknown targets, fixed-budget one-off crawls, and pipelines where per-page cost predictability matters more than raw cheapness.

The sequence that works: pilot against your real targets at production concurrency, log outcomes per request, compute effective cost per successful page under both models, then negotiate. Teams that skip the measurement step end up paying a premium for risk they don’t have — or discovering at 40% failure rate that the flat per-request invoice was never the real cost. Measure first. The invoice follows the logs, not the other way around.