Async Scraping at Scale: Jobs, Batches, Webhooks

Async Scraping at Scale: Jobs, Batches, and Webhooks

Synchronous POST /api/v1/scrape is the right default when a page returns in a few seconds and you want the HTML in the same HTTP response. It stops being a good fit when a render can take tens of seconds, when you need hundreds or thousands of URLs, or when you do not want a long-lived client connection sitting open while workers do the work.

FineData’s async API under /api/v1/async turns that into a job workflow: submit work, get an id immediately, then either poll for status or receive a webhook when the job finishes. This guide covers the real endpoints, the status values the gateway returns, polling with backoff, webhook authenticity checks, batches of up to 100 URLs, and client-side patterns that keep retries from creating duplicate work.

If you are new to the API, start with Getting Started with FineData. For how async jobs fit into a larger pipeline, see Scaling Web Scraping from 1K to 10M Pages and Building ETL Pipelines with Scraping. Full reference lives in the docs; plan limits are on pricing.

When Sync Is the Wrong Tool

Keep sync scrape for:

• One-off fetches and interactive tooling
• Pages that usually finish well under your HTTP client timeout
• Scripts where blocking until the response arrives is simpler than managing job state

Switch to async when:

• JS rendering, captcha solving, or multi-step browser actions make wall-clock time unpredictable
• You are enqueueing dozens to thousands of URLs and care about throughput more than a single round-trip
• Your caller (serverless function, CI job, webhook handler) should return quickly and process results later

Async requests accept the same scrape fields as sync — url, formats, only_main_content, extract_rules, extract_schema, extract_prompt, ai_content_mode, session_id, session_ttl, solve_captcha, stealth/proxy flags, and so on — plus two async-only fields: callback_url and callback_headers.

All examples below use Authorization: Bearer <API_KEY> and training URLs such as https://books.toscrape.com, https://quotes.toscrape.com, https://httpbin.org/html, and https://example.com.

Job Lifecycle

POST /api/v1/async/scrape
        │
        ▼
   job_id + status=pending
        │
        ▼
   status=processing   (worker picked up the job)
        │
        ├──────────────► status=completed  (+ result)
        ├──────────────► status=failed     (+ error)
        └──────────────► status=cancelled  (only if you cancel while pending)

Statuses come from the gateway’s JobStatus enum and the public response models:

Status	Meaning
pending	Queued, not started
processing	A worker is running the job
completed	Finished; result is available
failed	Finished with an error; check error
cancelled	Cancelled while still pending

There is also an internal expired state for TTL edge cases; list filters document the five statuses above.

Submit a Job

curl -sS -X POST "https://api.finedata.ai/api/v1/async/scrape" \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "formats": ["markdown"],
    "only_main_content": true,
    "use_js_render": false
  }'

Create response (AsyncJobResponse):

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "url": "https://books.toscrape.com/",
  "created_at": "2026-08-10T04:00:00",
  "estimated_completion": "2026-08-10T04:00:30"
}

Fields: job_id, status, url, created_at, and optional estimated_completion.

Poll for Status

curl -sS \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  "https://api.finedata.ai/api/v1/async/jobs/550e8400-e29b-41d4-a716-446655440000"

A completed AsyncJobStatusResponse includes (among other fields) job_id, status, url, method, created_at, started_at, completed_at, scrape option echoes, result, error, attempts, and tokens_used. On failure, read error; on success, read result (and any processed data when you requested formats / extract options).

Polling with Exponential Backoff and Jitter

Fixed sleep(1) loops hammer the API and waste client CPU. Use exponential backoff with jitter: start short, grow the wait, add randomness so many clients do not align on the same tick, and stop when the job is terminal.

import os
import random
import time
import requests

API = "https://api.finedata.ai/api/v1/async"
HEADERS = {
    "Authorization": f"Bearer {os.environ['FINEDATA_API_KEY']}",
    "Content-Type": "application/json",
}
TERMINAL = {"completed", "failed", "cancelled"}

def submit(url: str) -> str:
    r = requests.post(
        f"{API}/scrape",
        headers=HEADERS,
        json={"url": url, "formats": ["markdown"]},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()["job_id"]

def poll(job_id: str, *, base=1.0, cap=30.0, max_wait=600.0) -> dict:
    deadline = time.monotonic() + max_wait
    delay = base
    while time.monotonic() < deadline:
        r = requests.get(f"{API}/jobs/{job_id}", headers=HEADERS, timeout=30)
        r.raise_for_status()
        job = r.json()
        if job["status"] in TERMINAL:
            return job
        # full jitter: sleep in [0, delay]
        time.sleep(random.uniform(0, delay))
        delay = min(cap, delay * 2)
    raise TimeoutError(f"job {job_id} still not terminal after {max_wait}s")

job_id = submit("https://quotes.toscrape.com/")
job = poll(job_id)
if job["status"] == "completed":
    print(job.get("result"))
else:
    print("failed:", job.get("error"), "attempts:", job.get("attempts"))

Treat 429 / transient 5xx on the poll request itself as retryable with the same backoff. Do not treat a job failed status as “retry the GET forever” — that job is done; enqueue a new one if you want another attempt (see patterns below).

Webhooks Instead of Polling

Pass callback_url (and usually callback_headers) at submit time. When the job reaches completed or failed, the gateway POSTs a JSON payload to your URL and includes any custom headers you set.

import secrets

callback_token = secrets.token_urlsafe(32)

r = requests.post(
    f"{API}/scrape",
    headers=HEADERS,
    json={
        "url": "https://httpbin.org/html",
        "formats": ["markdown"],
        "callback_url": "https://hooks.example.com/finedata",
        "callback_headers": {
            "X-Callback-Token": callback_token,
        },
    },
    timeout=30,
)
print(r.json()["job_id"])
# Persist callback_token next to job_id in your store so the receiver can verify it.

Webhook body fields sent by the service:

{
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "completed",
  "url": "https://httpbin.org/html",
  "completed_at": "2026-08-10T04:01:12",
  "result": {},
  "error": null,
  "attempts": 1,
  "tokens_used": 3
}

Delivery retries a few times with backoff if your endpoint returns an HTTP error. Still design the receiver to be idempotent: the same job_id may arrive more than once.

Verify the Callback — Do Not Trust Blind POSTs

Anyone who can guess or discover your public webhook URL can POST a fake “completed” payload. FineData does not attach a platform HMAC to async callbacks; you authenticate the caller with a secret you put in callback_headers at submit time (for example X-Callback-Token) and check that header on receipt.

Requirements for a safe receiver:

1. Compare the token with a constant-time equality check against the value you stored for that job (or a global webhook secret).
2. Return 200 quickly; do heavy work in a background task/queue.
3. Ignore or reject payloads that fail the check — do not mutate your database because “a POST arrived.”

Minimal FastAPI receiver:

import hmac
import os
from fastapi import BackgroundTasks, FastAPI, Header, HTTPException, Request

app = FastAPI()
EXPECTED = os.environ["CALLBACK_TOKEN"]  # same value you put in callback_headers

def process_job(payload: dict) -> None:
    # Persist result / enqueue downstream ETL here.
    print(payload["job_id"], payload["status"])

@app.post("/finedata")
async def finedata_webhook(
    request: Request,
    background_tasks: BackgroundTasks,
    x_callback_token: str | None = Header(default=None),
):
    if not x_callback_token or not hmac.compare_digest(x_callback_token, EXPECTED):
        raise HTTPException(status_code=401, detail="invalid callback token")
    payload = await request.json()
    background_tasks.add_task(process_job, payload)
    return {"ok": True}

Minimal Flask receiver:

import hmac
import os
from flask import Flask, jsonify, request

app = Flask(__name__)
EXPECTED = os.environ["CALLBACK_TOKEN"]

@app.post("/finedata")
def finedata_webhook():
    token = request.headers.get("X-Callback-Token", "")
    if not hmac.compare_digest(token, EXPECTED):
        return jsonify({"error": "invalid callback token"}), 401
    payload = request.get_json(force=True, silent=True) or {}
    # Hand off to a worker/queue in production; keep the HTTP path short.
    app.logger.info("job=%s status=%s", payload.get("job_id"), payload.get("status"))
    return jsonify({"ok": True}), 200

Point callback_url at an HTTPS endpoint you control. FineData validates callback hosts server-side; you still own token verification and idempotent handling.

Batch: Up to 100 URLs

When you have a fixed list of URLs that share similar options, submit them together:

curl -sS -X POST "https://api.finedata.ai/api/v1/async/batch" \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      {"url": "https://books.toscrape.com/", "formats": ["markdown"]},
      {"url": "https://quotes.toscrape.com/", "formats": ["markdown"]},
      {"url": "https://example.com/", "formats": ["markdown"]}
    ],
    "callback_url": "https://hooks.example.com/finedata-batch"
  }'

Create response (BatchJobResponse): batch_id, job_ids, total_jobs, status, created_at. The hard limit is 100 requests per batch (schema max_length=100 and the router reject the same).

Progress:

curl -sS \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  "https://api.finedata.ai/api/v1/async/batch/$BATCH_ID?include_results=true"

BatchJobStatusResponse fields: batch_id, status, total_jobs, completed_jobs, failed_jobs, created_at, completed_at, and optional results when include_results=true.

Batch-level statuses:

Status	Meaning
pending	Just created / not started
processing	Jobs still running
completed	All jobs succeeded
partial	Finished with a mix of successes and failures
failed	All jobs failed

Note: BatchScrapeRequest accepts a batch-level callback_url, but not callback_headers. For token-checked delivery, prefer per-job POST /async/scrape with callback_headers, or protect the batch webhook with network controls plus your own shared secret at the edge if you terminate TLS yourself.

Batch vs a Queue of Single Jobs

Use batch when:

• You have ≤100 URLs ready at once
• You want one batch_id for progress (completed_jobs / failed_jobs)
• Options are mostly homogeneous

Use single jobs (your own queue calling POST /async/scrape) when:

• The catalog is larger than 100 and you shard into many submits over time
• Each URL needs different callbacks, sessions, or extract settings
• You need client-side idempotency keys mapped one-to-one to job_id
• You want callback_headers for webhook authentication

A common production shape: your scheduler pops work units, submits async jobs (or chunks of 100 as batches), stores job_id/batch_id, and reconciles via list + webhooks.

Listing Jobs for Reconciliation

curl -sS \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  "https://api.finedata.ai/api/v1/async/jobs?limit=50&offset=0&status=failed"

GET /api/v1/async/jobs returns AsyncJobListResponse: jobs, total, limit, offset. Query params: limit (1–100, default 50), offset, and status (pending, processing, completed, failed, cancelled).

Use listing when:

• A webhook may have been missed and you need to find stuck processing / unfinished pending work
• You want a daily reconcile: every local task key should map to a terminal FineData job
• You are draining failed jobs into a retry topic

Optional: DELETE /api/v1/async/jobs/{job_id} cancels a job only while it is still pending. Jobs already processing run to completion.

Client Patterns That Survive Real Failures

Idempotency on Your Side

The async create endpoints do not take a FineData-managed idempotency key. If your HTTP POST times out after the server accepted the job, a naive retry creates a second job for the same URL.

Pattern:

1. Generate a stable client key per work unit (hash of URL + extract config + crawl run id).
2. Before submit, look up that key in Redis/Postgres.
3. If a job_id already exists, poll or wait for webhook — do not POST again.
4. On a successful create response, store client_key → job_id atomically.

import hashlib
import json
import redis

r = redis.Redis.from_url(os.environ["REDIS_URL"])

def client_key(url: str, body: dict) -> str:
    raw = json.dumps({"url": url, **body}, sort_keys=True)
    return "scrape:" + hashlib.sha256(raw.encode()).hexdigest()

def submit_once(url: str, body: dict) -> str:
    key = client_key(url, body)
    existing = r.get(key)
    if existing:
        return existing.decode()
    resp = requests.post(
        f"{API}/scrape",
        headers=HEADERS,
        json={"url": url, **body},
        timeout=30,
    )
    resp.raise_for_status()
    job_id = resp.json()["job_id"]
    # SET NX so a concurrent submit does not overwrite a winner.
    if r.set(key, job_id, nx=True, ex=86400):
        return job_id
    return r.get(key).decode()

Failed Jobs: Bounded Retries, New Job IDs

A failed job is terminal. Retry by creating a new job (new job_id), not by polling the old one forever. Cap retries (for example three attempts), classify errors, and back off between submits. Log attempts and tokens_used from the failed response so you can tell infrastructure flakes from persistent target errors.

Rate-Limit Friendly Clients

• Prefer webhooks for high volume; poll only for small fleets or reconcile passes
• Cap concurrent submits from your side; do not open thousands of create calls in one burst
• Honor 429 with Retry-After when present, otherwise exponential backoff
• Use batch for dense URL lists so you pay one create round-trip per up to 100 URLs
• Keep list/poll traffic on a longer interval once most jobs are terminal

Wiring into ETL

Typical flow: discover URLs → enqueue work units → submit async jobs/batches → verify webhooks → write markdown/extract JSON to storage → mark the client key done. Same queue idea as the scaling guide; FineData owns the fetch workers. Agents that prefer tools over raw HTTP can use the MCP protocol guide; for long runs and reconciliation, the REST async routes remain the control plane.

Minimal End-to-End Checklist

1. Decide sync vs async (timeout and volume).
2. Submit; capture job_id or batch_id.
3. Store a client idempotency key → id mapping before retries.
4. Poll with jittered backoff, or set callback_url + secret callback_headers.
5. Verify the callback token; ack 200; process in the background.
6. Reconcile with GET /api/v1/async/jobs (and batch status) on a schedule.
7. Retry failures as new jobs with a bounded counter.

Async scraping is an explicit job state machine: use the statuses the API returns, authenticate webhooks, and keep idempotency in your datastore so long renders and multi-thousand URL crawls do not require holding HTTP connections open for the entire fetch.