Fix Scraped Data That Lands in the Wrong Fields

Symptom Check: Why ‘Valid’ JSON Can Still Be Wrong-Field Data

Your scraper returns JSON. It parses. Your ingestion layer accepts it. And yet the dashboard shows a product whose description is “$49.99” and whose price is “Soft cotton tee, machine washable.” Nothing crashed. Nothing threw. The data is structurally fine and semantically garbage.

This is the worst failure mode in a scraping pipeline, because every downstream system you own is built to trust schema-valid input. A parse error gets caught in seconds. A mapping error propagates until a human notices that the average price on your price-tracking dashboard is now a float derived from a product description.

Here’s what it looks like side by side. Both records come from store.example.com, and both pass json.loads() without complaint:

Field	Correct record	Mis-mapped record
title	"Classic Canvas Sneaker"	"Classic Canvas Sneaker"
price	49.99	"Breathable mesh upper, ideal for daily wear"
description	"Breathable mesh upper, ideal for daily wear"	"$49.99"
currency	"USD"	"USD"
Validation	Passes	Also passes

A quick sanity check separates a mapping bug from a parsing bug. If the values are all present but sitting in the wrong keys, you have a mapping bug. If values are truncated, mangled, or missing entirely, you have a parsing bug. This snippet diffs expected value shapes against actual values and flags records where a field’s content doesn’t match its contract:

import json, re

PRICE_RE = re.compile(r"^\d+(\.\d{1,2})?$")

def flag_suspect_records(path):
    suspects = []
    with open(path) as f:
        for i, line in enumerate(f):
            rec = json.loads(line)
            price = str(rec.get("price", ""))
            desc = str(rec.get("description", ""))
            # price must be numeric; description must not look like a price
            if not PRICE_RE.match(price):
                suspects.append((i, "price not numeric", rec))
            elif PRICE_RE.match(desc):
                suspects.append((i, "description looks like a price", rec))
    return suspects

If you’re on the command line and the dump is small, jq does the same job faster:

jq -c 'select((.price | type) != "number" or (.description | test("^\\d+\\.?\\d*$")))' products.jsonl

Either way, the output tells you two things: how widespread the problem is, and whether the mis-mapping is consistent (one selector broke) or scattered (page templates differ per product).

Trace the Root Cause: How Fragile CSS Selectors Shift Columns

Nine times out of ten, the culprit is a positional selector. Someone wrote .product-row span:nth-child(3) because at the time, the third span in a product row was the price. That assumption held for exactly as long as the DOM never changed.

Here’s a stripped-down product card from store.example.com:

<div class="product-row">
  <span class="badge">-20%</span>
  <span class="title">Classic Canvas Sneaker</span>
  <span class="price">$49.99</span>
  <span class="desc">Breathable mesh upper, ideal for daily wear</span>
</div>

<div class="product-row">
  <!-- no discount badge on this one -->
  <span class="title">Minimal Leather Belt</span>
  <span class="price">$29.00</span>
  <span class="desc">Full-grain leather, brass buckle</span>
</div>

The second card has no badge. span:nth-child(3) returns the description for the first card and the price for the second. Your scraper happily writes both into the same field. The bug isn’t triggered by a site redesign — it’s triggered by ordinary content variance that existed on the page the whole time.

The bad-versus-better pair:

# BAD: position-based. Breaks the moment a product lacks a badge,
# or the site inserts a "New" or "Free shipping" label.
price = row.select_one("span:nth-child(3)").text
desc  = row.select_one("span:nth-child(4)").text

# BETTER: class-based. Survives missing badges and reordering.
price = row.select_one("span.price").text
desc  = row.select_one("span.desc").text

Class selectors are better, but classes get renamed during redesigns too. Before you fix anything, open the page and look for the hooks that survive visual changes: itemprop microdata, data-* attributes, ARIA labels, structured data embedded in <script type="application/ld+json">. If the site publishes JSON-LD, skip DOM scraping for those fields entirely — the price is already in a typed JSON block.

One more thing worth checking before you rewrite anything: is the mis-mapping actually selector drift, or did the site ship a new template variant? Load two failing URLs and one working URL, diff their DOM structure, and confirm. Fixing selectors against the wrong diagnosis wastes an afternoon.

Fix 1: Anchor Selectors to Stable Attributes Instead of Position

Rewrite the extraction to bind each field to a semantic anchor. Attribute selectors like [itemprop="price"] and [data-testid="..."] outlive class renames because they exist to describe meaning, not styling — and test IDs specifically exist so that the site’s own test suite breaks when someone removes them, which makes them the stickiest hooks you’ll find.

from bs4 import BeautifulSoup
import re

PRICE_CLEAN = re.compile(r"[^\d.]")

def extract_product(row: BeautifulSoup) -> dict:
    # itemprop microdata: survives class renames and layout shifts
    price_el = row.select_one('[itemprop="price"]')
    price = None
    if price_el:
        raw = price_el.get("content") or price_el.text
        price = float(PRICE_CLEAN.sub("", raw)) if raw else None

    # data-testid hooks: stable because the site's own tests depend on them
    title = row.select_one('[data-testid="product-title"]')
    desc  = row.select_one('[data-testid="product-description"]')

    return {
        "title": title.get_text(strip=True) if title else None,
        "price": price,
        "description": desc.get_text(strip=True) if desc else None,
        "currency": (row.select_one('[itemprop="priceCurrency"]') or {}).get("content", "USD") if row.select_one('[itemprop="priceCurrency"]') else "USD",
    }

How three selector strategies hold up against a simulated DOM change (badge removed from some cards, classes renamed in a redesign):

Selector	Failure mode	Survives missing badge	Survives class rename
span:nth-child(3)	Silent column shift	No	No
span.price	Returns None (loud)	Yes	No
[itemprop="price"]	Returns None (loud)	Yes	Yes

Notice the pattern: better selectors don’t magically succeed — they fail visibly. A selector that returns None is a bug you catch in testing. A selector that returns the wrong element is a bug your customers catch. Prefer the first kind, always.

My honest opinion here: I’d rather have a scraper that throws on 5% of pages than one that silently produces plausible garbage on 5% of pages. Silent partial success is the expensive failure. If you want more on where positional selectors break relative to model-based extraction, I covered the failure modes in CSS Selectors vs LLM Extraction: What Breaks First.

Fix 2: Enforce a JSON Schema So Bad Mappings Fail Loudly

Fixed selectors reduce the bug rate. A schema catches whatever slips through. The two work together: selectors are prevention, schema validation is detection.

Define the contract for a product record:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Product",
  "type": "object",
  "required": ["title", "price", "currency"],
  "properties": {
    "title": { "type": "string", "minLength": 2, "maxLength": 300 },
    "price": { "type": "number", "minimum": 0.01, "maximum": 100000 },
    "description": { "type": ["string", "null"], "maxLength": 5000 },
    "currency": { "type": "string", "pattern": "^[A-Z]{3}$" }
  },
  "additionalProperties": false
}

Then validate at extraction time, before anything touches your database:

from jsonschema import Draft202012Validator

validator = Draft202012Validator(schema)

def validate_record(rec: dict):
    errors = sorted(validator.iter_errors(rec), key=lambda e: e.path)
    if errors:
        # Reject: price is a string with a currency symbol,
        # or a description-length string landed in the price field
        raise ValueError(f"Mapping contract violated: {[e.message for e in errors]}")

# This one fails loudly:
validate_record({
    "title": "Classic Canvas Sneaker",
    "price": "Breathable mesh upper, ideal for daily wear",  # wrong field
    "description": "$49.99",                                   # wrong field
    "currency": "USD"
})
# -> ValueError: Mapping contract violated:
#    "'Breathable mesh upper...' is not of type 'number'"

The maximum: 100000 constraint is doing quiet work here — if a selector grabs a product ID or a review count into the price field, the value often lands far outside a plausible price range and gets rejected even though it’s numeric. Type checks catch string contamination; range checks catch numeric contamination. You need both.

One trade-off to accept: strict validation means some legitimate records get rejected. A store selling industrial equipment might have prices above your cap; a marketplace might legitimately have 10-character currency strings. Tune the schema to your actual domain, and route rejects to a dead-letter queue rather than dropping them — the rejects are your best signal about which selectors are drifting.

Fix 3: Repair Mis-Mapped Records with a Targeted Re-Extraction Script

Fixing the scraper doesn’t fix the 4,800 records already in your dump. Don’t re-scrape everything — that’s wasted spend and wasted time. Re-fetch only the records that fail the schema, using the corrected extraction.

This script scans the dump, flags failures, and re-extracts just those URLs through the scrape API with the repaired selectors:

import json, requests
from jsonschema import Draft202012Validator

API = "https://api.finedata.ai"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}
validator = Draft202012Validator(schema)

def repair(dump_path: str, out_path: str):
    flagged, repaired, manual = [], [], []

    with open(dump_path) as f:
        records = [json.loads(line) for line in f]

    for rec in records:
        if not list(validator.iter_errors(rec)):
            continue
        flagged.append(rec)

    for rec in flagged:
        resp = requests.post(f"{API}/api/v1/scrape", headers=HEADERS, json={
            "url": rec["source_url"],
            "extract_rules": {
                "title": '[data-testid="product-title"]',
                "price": '[itemprop="price"]',
                "description": '[data-testid="product-description"]',
                "currency": '[itemprop="priceCurrency"]'
            },
            "formats": ["text"],
            "use_js_render": True,
            "only_main_content": True
        })
        resp.raise_for_status()
        fresh = resp.json().get("data", {})

        if list(validator.iter_errors(fresh)):
            manual.append(rec)   # still broken: page-level oddity, needs eyes
        else:
            fresh["source_url"] = rec["source_url"]
            repaired.append(fresh)

    with open(out_path, "w") as out:
        for rec in repaired:
            out.write(json.dumps(rec) + "\n")
    return len(flagged), len(repaired), len(manual)

Typical run against a real dump:

[repair] scanning products_2025-... no —
[repair] scanning products.jsonl: 4800 records
[repair] schema failures: 312 (6.5%)
[repair] re-fetching 312 URLs with corrected extract_rules
[repair] auto-corrected: 298
[repair] routed to manual review: 14 (validation still failing)
[repair] done in 214s

Six and a half percent contamination is about what you’d expect from a single broken selector that only fires on badge-less product cards. The 14 leftovers usually fall into two buckets: pages that changed template entirely, and products that were delisted between your first scrape and the repair pass. Both need a human, and both are worth a look — the delisted-product pattern tells you something about inventory churn.

A related decision — whether retries belong in your client or in the service — has its own trade-offs, which I broke down in Client-Side vs Service-Side Retries for Scraping Failures. Short version: for a one-off repair pass, client-side is fine.

Fix 4: Guardrails for LLM-Based Extraction — Prompt Constraints and Output Contracts

If you use prompt-based extraction instead of selectors, the same disease shows up with different symptoms. The model doesn’t shift columns because the DOM changed — it shifts them because your prompt never pinned down the contract. Ask it to “extract the product info” and it will decide, per page, whether “price” means the current price, the strikethrough price, or the per-unit price. That decision will not be consistent across pages.

Here’s a constrained prompt that removes the model’s discretion:

Extract product data from this page. Return ONLY a JSON object
with exactly these keys:

- "title": string, the product name from the main heading.
  Max 300 characters.
- "price": number, the current buy price in the page's currency.
  Digits and decimal point only — no currency symbol, no
  strikethrough/compare-at price. If only a price range is shown,
  use the lower bound.
- "compare_at_price": number or null. The crossed-out original
  price. null if not shown.
- "currency": string, ISO 4217 three-letter code. If not stated,
  null.
- "description": string or null. The main product description
  paragraph. null if not found.

Rules:
- If a value is not on the page, use null. NEVER guess, NEVER
  copy a value from another field.
- Do not add, rename, or omit keys.
- Output raw JSON. No markdown, no commentary.

Same page, two prompts, two outcomes:

import requests

API = "https://api.finedata.ai"
HEADERS = {"Authorization": "Bearer fd_your_api_key"}

# Vague prompt — model improvises key semantics
vague = requests.post(f"{API}/api/v1/scrape", headers=HEADERS, json={
    "url": "https://store.example.com/products/canvas-sneaker",
    "extract_prompt": "Get the product details from this page.",
    "formats": ["markdown"]
}).json()

# Constrained prompt — same page, explicit contract
strict = requests.post(f"{API}/api/v1/scrape", headers=HEADERS, json={
    "url": "https://store.example.com/products/canvas-sneaker",
    "extract_prompt": """Extract product data. Return ONLY JSON with keys:
title (string), price (number, current buy price, no symbol),
compare_at_price (number or null), currency (ISO 4217 or null),
description (string or null). Use null when absent — never guess,
never copy between fields.""",
    "formats": ["markdown"]
}).json()

// vague prompt result — plausible, wrong
{ "name": "Classic Canvas Sneaker",
  "price": "was $61.99", "original_price": 49.99,
  "details": "Breathable mesh upper" }

// constrained prompt result — matches the contract
{ "title": "Classic Canvas Sneaker",
  "price": 49.99, "compare_at_price": 61.99,
  "currency": "USD",
  "description": "Breathable mesh upper, ideal for daily wear" }

The vague output is the more dangerous one, because it looks reasonable at a glance. Note what the model did: it grabbed the strikethrough price as original_price, stuffed a formatted string into price, and renamed title to name. Every one of those is a mapping decision you never approved.

And here’s the part people miss: the prompt alone is not enough. Run the model’s output through the same JSON Schema from Fix 2. A prompt is a request; a schema is a gate. When the model inevitably returns "price": "$49.99" as a string on some page, the schema rejects it and you re-run that one page instead of shipping the contamination. For more on schema-driven extraction generally, see From HTML to JSON: Schema-Driven Extraction.

Regression-Proof the Pipeline: A Golden-Dataset Check That Catches Silent Drift

Everything above fixes today’s bug. This section prevents next month’s. Sites redesign, and when they do, your selectors will drift again — the only question is whether you find out from a test or from a customer.

Save three representative pages from example.com as fixtures (pick deliberately: one normal product, one missing an optional element, one edge case like a sale price), and assert known ground-truth values:

# test_golden_dataset.py
import json
from pathlib import Path
import pytest
from myscraper import extract_product  # your fixed extractor

FIXTURES = Path(__file__).parent / "fixtures"

CASES = [
    {
        "file": "normal_product.html",
        "expected": {"title": "Classic Canvas Sneaker",
                     "price": 49.99, "currency": "USD"},
    },
    {
        "file": "no_badge_product.html",
        "expected": {"title": "Minimal Leather Belt",
                     "price": 29.00, "currency": "USD"},
    },
    {
        "file": "sale_product.html",
        "expected": {"title": "Wool Overcoat",
                     "price": 189.00, "currency": "USD"},
    },
]

@pytest.mark.parametrize("case", CASES)
def test_extraction_matches_ground_truth(case):
    html = (FIXTURES / case["file"]).read_text()
    result = extract_product(html)
    for field, expected in case["expected"].items():
        assert result[field] == expected, (
            f"{field} drifted on {case['file']}: "
            f"expected {expected!r}, got {result[field]!r}"
        )

Then run it on a schedule, because a test that only runs when you push code won’t catch a redesign that happens while you’re not touching the repo:

# .github/workflows/golden-dataset.yml
name: golden-dataset
on:
  schedule:
    - cron: "0 6 * * *"   # daily
  workflow_dispatch:

jobs:
  drift-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt pytest
      - name: Refresh live fixtures
        run: python scripts/refresh_fixtures.py  # re-fetches the 3 URLs
      - name: Run golden-dataset assertions
        run: pytest test_golden_dataset.py -v --tb=short

The refresh step matters. If your fixtures are two years stale, you’re testing against a page that no longer exists. The workflow re-fetches the three URLs, saves fresh HTML, and runs the assertions against ground truth — so a redesign that breaks [itemprop="price"] fails the build within a day instead of surfacing as a support ticket three weeks later.

Keep the fixture set small. Three to five pages is plenty; fifty turns a ten-second check into a maintenance burden you’ll disable. The goal is a tripwire, not a mirror of production.

Wrap-Up

Wrong-field data is a mapping bug wearing the costume of valid JSON, and it survives precisely because nothing in your pipeline is allowed to complain. The defense is layered, and each layer catches what the previous one misses:

1. Anchor selectors to semantic attributes (itemprop, data-testid) so ordinary content variance can’t shift columns.
2. Validate against a JSON Schema at extraction time so contamination fails loudly, in seconds, at the source.
3. Repair existing dumps selectively — re-fetch only schema failures, not the whole corpus.
4. Constrain prompt-based extraction with exact key names, types, and a null-if-absent rule, then gate the output through the same schema.
5. Run a small golden-dataset check on a schedule so the next redesign trips a build failure instead of your customers.

Gotchas to keep in mind: schemas need domain-appropriate bounds or you’ll reject legitimate records; prompt contracts without schema gates are suggestions, not guarantees; and never auto-repair into your primary store — write corrected records to a staging path and diff before replacing. If the mis-mapping rate is high and consistent across a whole site, that’s usually not drift at all — it’s a new page template, and you want to fix the extractor once rather than repair records forever.

Next step if you want to go deeper on the pipeline side: Parse at Fetch Time vs Store Raw HTML covers whether to extract at scrape time or keep raw HTML so you can re-parse historical dumps without re-fetching — which would have made the repair pass in Fix 3 a pure CPU job instead of 312 network requests.