From HTML to JSON: Schema-Driven Extraction

From HTML to JSON: Schema-Driven Extraction

Scraping is rarely the end of the pipeline. Downstream jobs want rows: title, price, availability, author — typed fields you can load into a warehouse or feed an ETL step. Parsing HTML by hand works until the markup drifts. FineData exposes three extraction modes on the same scrape call so you can pick the right tool per field, not one blunt approach for every page.

This guide walks through extract_schema, extract_prompt, and extract_rules, when to use ai_content_mode, how formats CSV/XLSX export works, and how to validate what the model returns. All examples hit training sandboxes (books.toscrape.com, quotes.toscrape.com) or store.example.com — nothing you should aim a production scraper at until you have permission and a measured quality loop.

If you are new to the API, start with Getting Started with FineData. For how extracted JSON fits into batch jobs, see Building ETL Pipelines from Scraped Data.

The Three Extraction Tools

Every sync scrape goes to POST https://api.finedata.ai/api/v1/scrape with Authorization: Bearer <API_KEY>. Extraction fields ride on that request. You can combine selector rules with AI modes when it helps; cost and failure modes differ.

Mode	What you supply	Best for	Cost note	Failure mode
extract_rules	CSS/XPath map (title → h1)	Stable markup, known selectors	No AI surcharge	Empty/wrong field when DOM changes
extract_schema	JSON Schema object	Typed, nested records you will validate	+5 tokens	Incomplete or schema-skewed JSON
extract_prompt	Natural-language instruction	Ad-hoc fields, exploratory pulls	+5 tokens	Shape drift between runs

Choose extract_rules when the DOM is stable and you care about determinism more than resilience. Catalog grids with consistent classes are the classic case.

Choose extract_schema when you need a contract: required keys, nested arrays, numeric types. The model fills the schema; your client can reject invalid payloads with jsonschema before they hit the warehouse.

Choose extract_prompt when you are prototyping or the field set is fuzzy (“summarize the product bullets”). Prefer promoting a working prompt into a schema once the shape stabilizes — schemas keep pipelines honest.

AI extraction (extract_schema or extract_prompt) adds +5 tokens on a successful request. FineData bills on pay-for-success: failed scrapes do not consume those tokens. See current rates on Pricing.

Content Scope: ai_content_mode and only_main_content

Two knobs control how much of the page the extractor sees.

• ai_content_mode: "full" (default) or "main". Applies to AI extraction (extract_schema / extract_prompt).
  • "full" — product pages, listings, anything where price, stock, or structured blocks live outside the main article column.
  • "main" — articles and long prose where Readability-style main content cuts nav/footer noise and usually means a smaller prompt context.
• only_main_content: boolean for markdown / text formats. Same idea for human-readable output, independent of AI extraction.

Rule of thumb: listings and product cards → ai_content_mode: "full". Blog posts and documentation → "main". If you also want clean markdown for logging, set only_main_content: true with formats: ["markdown"].

Example 1: Deterministic Rules on Books to Scrape

https://books.toscrape.com is a public training storefront. Selectors stay put, so extract_rules is the right first tool.

curl -sS https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "formats": ["markdown"],
    "extract_rules": {
      "title": "h1",
      "book_titles": {
        "selector": "article.product_pod h3 a",
        "type": "list",
        "output": "@title"
      },
      "book_links": {
        "selector": "article.product_pod h3 a",
        "type": "list",
        "output": "@href"
      },
      "prices": {
        "selector": "article.product_pod .price_color",
        "type": "list"
      }
    }
  }'

Rule shapes:

• Simple string: "title": "h1" → text of the first match (type text by default).
• Advanced object: selector, optional type (text | html | list), optional output (text content, or @attr such as @href / @title).

Python equivalent:

import os
import requests

API = "https://api.finedata.ai/api/v1/scrape"
HEADERS = {
    "Authorization": f"Bearer {os.environ['FINEDATA_API_KEY']}",
    "Content-Type": "application/json",
}

resp = requests.post(
    API,
    headers=HEADERS,
    json={
        "url": "https://books.toscrape.com/",
        "formats": ["markdown"],
        "extract_rules": {
            "title": "h1",
            "prices": {
                "selector": "article.product_pod .price_color",
                "type": "list",
            },
            "book_links": {
                "selector": "article.product_pod h3 a",
                "type": "list",
                "output": "@href",
            },
        },
    },
    timeout=60,
)
resp.raise_for_status()
data = resp.json()
print(data.get("extract"))

Parsed fields land under the response extract object. When markup changes, fix the selectors — there is no model improvisation to hide a broken CSS path.

Example 2: Schema-Driven AI Extraction

Same catalog, but you want typed JSON without maintaining every selector. Pass a JSON Schema in extract_schema and keep ai_content_mode on "full" so listing cards are in view.

import os
import requests

API = "https://api.finedata.ai/api/v1/scrape"
HEADERS = {
    "Authorization": f"Bearer {os.environ['FINEDATA_API_KEY']}",
    "Content-Type": "application/json",
}

schema = {
    "type": "object",
    "properties": {
        "books": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "title": {"type": "string"},
                    "price": {"type": "string"},
                    "in_stock": {"type": "boolean"},
                },
                "required": ["title", "price"],
            },
        }
    },
    "required": ["books"],
}

resp = requests.post(
    API,
    headers=HEADERS,
    json={
        "url": "https://books.toscrape.com/",
        "formats": ["markdown"],
        "extract_schema": schema,
        "ai_content_mode": "full",
    },
    timeout=90,
)
resp.raise_for_status()
payload = resp.json()
print(payload.get("extract"))

curl variant:

curl -sS https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "formats": ["markdown"],
    "ai_content_mode": "full",
    "extract_schema": {
      "type": "object",
      "properties": {
        "books": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "title": {"type": "string"},
              "price": {"type": "string"},
              "in_stock": {"type": "boolean"}
            },
            "required": ["title", "price"]
          }
        }
      },
      "required": ["books"]
    }
  }'

For article-like pages (quotes, blog posts), switch to "main":

resp = requests.post(
    API,
    headers=HEADERS,
    json={
        "url": "https://quotes.toscrape.com/",
        "formats": ["markdown"],
        "only_main_content": True,
        "ai_content_mode": "main",
        "extract_schema": {
            "type": "object",
            "properties": {
                "quotes": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "text": {"type": "string"},
                            "author": {"type": "string"},
                            "tags": {
                                "type": "array",
                                "items": {"type": "string"},
                            },
                        },
                        "required": ["text", "author"],
                    },
                }
            },
            "required": ["quotes"],
        },
    },
    timeout=90,
)

Example 3: Natural-Language extract_prompt

Useful for exploration or one-off field lists. The tradeoff is a looser output shape than a schema.

curl -sS https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com/",
    "formats": ["markdown"],
    "ai_content_mode": "main",
    "extract_prompt": "Extract each quote text, author name, and tags. Return JSON with a quotes array."
  }'

Once the shape settles, copy it into extract_schema and drop the free-form prompt for production jobs.

Tabular Export: csv and xlsx

When you request AI extraction, you can also ask for tabular formats. formats may include csv and/or xlsx; both require extract_prompt or extract_schema. The API flattens the extraction result into rows suitable for spreadsheets or warehouse staging.

resp = requests.post(
    API,
    headers=HEADERS,
    json={
        "url": "https://books.toscrape.com/",
        "formats": ["csv", "xlsx"],
        "extract_schema": schema,
        "ai_content_mode": "full",
    },
    timeout=90,
)
body = resp.json()
# body may include csv string and/or xlsx payload depending on response shape
print(body.get("csv"))

Other format values on the same field: markdown, rawHtml, text, links, screenshot. Pick what the next stage needs; do not request every format “just in case.”

Raw Output Without a JSON Envelope

Set raw_output: true with exactly one entry in formats to receive the content bytes/string directly. Metadata moves to response headers: X-Tokens-Used, X-Request-Id, X-Status-Code.

curl -sS https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer $FINEDATA_API_KEY" \
  -H "Content-Type: application/json" \
  -D - \
  -o page.md \
  -d '{
    "url": "https://quotes.toscrape.com/",
    "formats": ["markdown"],
    "only_main_content": true,
    "raw_output": true
  }'

Use this when a downstream tool expects a file, not a nested JSON document. Pair it with AI extraction only when a single tabular format (csv or xlsx) is enough.

Validate, Retry, Fall Back

LLM extraction is probabilistic. Treat the API response as untrusted until it passes your schema:

1. Client-side jsonschema validation — load the same schema you sent (or a stricter one) and reject incomplete objects.
2. Retry with a tighter prompt or schema — add required, constrain enums, or clarify units (“price as a string including currency symbol”).
3. Fallback to extract_rules for stable fields — title, price CSS paths on training sites rarely move; let the model handle messy prose and keep selectors for anchors you must never lose.

import jsonschema

def validate_extract(extract, schema):
    """Raise if AI extract does not match the contract."""
    if extract is None:
        raise ValueError("missing extract payload")
    jsonschema.validate(instance=extract, schema=schema)

try:
    validate_extract(payload.get("extract"), schema)
except jsonschema.ValidationError as exc:
    # Retry with a clarified extract_prompt, or merge extract_rules for title/price
    print("invalid extract:", exc.message)

Do not invent accuracy percentages from blog posts. Measure on your own labeled sample: precision/recall per field, null rate, and how often retries recover. A small golden set from books.toscrape.com or your staging store.example.com pages is enough to catch regressions when you change schemas.

Putting It in a Pipeline

A practical pattern for catalog-style sites:

1. Scrape listing pages with extract_rules for links (type: "list", output: "@href").
2. Scrape detail pages with extract_schema + ai_content_mode: "full".
3. Validate JSON; on failure, retry once with a stricter schema, then fall back to selectors for critical fields.
4. Export accepted rows via formats: ["csv"] or push JSON into your ETL path.

That split keeps token spend on pages that need AI and keeps link discovery cheap. For cost framing of API vs DIY parsers over time, see Scraping API vs DIY. Broader product-page patterns (still using training or example.com targets in examples) are covered in Marketplace Product Data with Python. Free-tier limits for experiments live in FineData Free Tier. Full parameter reference: Docs.

Checklist

• Prefer extract_rules for stable, high-volume fields; extract_schema for typed contracts; extract_prompt for exploration.
• Use ai_content_mode: "full" on product/listing pages, "main" on articles.
• Add only_main_content when you want cleaner markdown/text.
• Request csv / xlsx only with AI extraction enabled.
• Validate every AI payload client-side; plan a selector fallback.
• Remember AI extraction is +5 tokens on success; failed requests are not charged under pay-for-success.

Schema-driven extraction does not remove the need for tests — it moves the maintenance surface from brittle CSS trees to schemas and validation you already understand. Start on a sandbox URL, lock a golden set, then promote the same request shape into production jobs with targets you are allowed to scrape.