Your Scraper Died in a Redesign: An Extraction Escalation

Introduction

Every pipeline has that one morning. The cron job runs, the logs fill up, and your dashboard shows a wall of None where prices used to be. HTTP status is 200. The page loads. Your selectors return nothing. The site shipped a redesign overnight, and your extraction layer — which worked flawlessly for months — is now parsing markup that no longer exists.

This post walks through an escalation ladder for exactly that situation: confirm the redesign is the culprit, diff the damage, then climb from cheap fixes (CSS selectors) to structural fixes (embedded structured data) to the expensive last resort (LLM prompt extraction). The goal is that the next redesign costs you hours, not weeks.

Symptoms of a Selector Collapse: How to Confirm It’s the Redesign and Not Your Code

Before you touch any extraction code, prove that the failure is on the site’s side. The most common false alarm is a bug you introduced yourself — a dependency upgrade, a proxy change, a typo pushed on Friday. Check three things in order.

First, fetch the page manually with curl and eyeball it. If the product title renders in a browser but your parser sees nothing, the markup changed. Second, check whether the failure is total or partial. A redesign usually kills entire field groups at once: price and availability die together because they lived in the same component. A bug in your code tends to break one field in isolation. Third, check the HTTP layer. A redesign often ships alongside a URL restructure, so old product URLs start 404ing or redirecting.

Here’s the diagnostic I run first — a minimal extraction against one known-good URL, logging what comes back:

import requests
from bs4 import BeautifulSoup

resp = requests.get(
    "https://store.example.com/product/123",
    headers={"User-Agent": "Mozilla/5.0 (compatible; pricebot/1.0)"},
    timeout=30,
)
soup = BeautifulSoup(resp.text, "html.parser")

# Pre-redesign selectors
title = soup.select_one("h1.product-title")
price = soup.select_one("div.price > span.amount")

print(f"status={resp.status_code}")
print(f"title={title.get_text(strip=True) if title else None}")
print(f"price={price.get_text(strip=True) if price else None}")

Before the redesign this printed title=Nova Running Shoe and price=$89.99. After it, both are None while the status stays 200. That combination — healthy HTTP, dead fields — is the signature of a markup change, not a network or auth problem.

Different failure signatures point to different causes. Don’t treat them all as “the selectors broke”:

Signature	What you observe	Likely cause
HTTP 200, all fields null	Page fetches fine, every selector misses	Markup restructure; your selectors are stale
404s or redirect chains on old URLs	Some jobs fail entirely, others succeed	URL scheme changed; you need a discovery refresh, not selector fixes
Content-type changed (e.g., text/html → application/json)	Your HTML parser chokes or returns garbage	The page is now a client-rendered shell fed by an embedded API

That third row matters more than people expect. If the content-type flipped, no amount of selector surgery will help — the product data now lives in a JSON payload inside a <script> tag, which is actually good news (more on that at rung 2).

One more check before escalating: confirm the redesign is live for everyone, not just for your proxy’s exit IP. Fetch the same URL from your laptop and from your scraper environment. If they differ, you’re looking at an A/B test or geo-targeted markup, and the fix is session pinning, not selector rewrites. I’ve written about keeping one exit IP across multi-step scrape requests here, and it applies directly to A/B-split diagnosis.

Snapshot the Damage: Diffing the Old and New DOM

Once you’ve confirmed a redesign, resist the urge to start rewriting selectors immediately. Ten minutes of diffing saves hours of guesswork, because it tells you exactly which selectors died and whether any of them survived.

If you archived raw HTML in your pipeline (and you should — I made the case for storing raw HTML over parse-at-fetch in a previous post), you already have the “before” side. If not, pull a snapshot from the Wayback Machine or any cached copy you have, then fetch the live page:

# Save the current (post-redesign) HTML
curl -s -A "Mozilla/5.0 (compatible; pricebot/1.0)" \
  "https://store.example.com/product/123" \
  -o post_redesign.html

# If you have an archived copy, normalize whitespace on both before diffing
# (HTML diffs are useless without normalization)
python - <<'EOF'
import re
for f in ("pre_redesign.html", "post_redesign.html"):
    html = open(f).read()
    html = re.sub(r"\s+", " ", html)
    open(f.replace(".html", "_norm.html"), "w").write(html)
EOF

diff pre_redesign_norm.html post_redesign_norm.html > dom_diff.txt
wc -l dom_diff.txt

The diff will be enormous — redesigns touch everything — so don’t read it line by line. Grep it for the class names your selectors depend on:

grep -o 'class="[^"]*price[^"]*"' post_redesign.html | sort | uniq -c
grep -o 'class="[^"]*product[^"]*"' post_redesign.html | sort | uniq -c

If price appears nowhere in the new markup, the class is gone and every selector built on it is dead. If it appears with a different parent, you may only need to adjust one level of the chain.

Here’s the typical before/after for a product card, abbreviated:

<!-- BEFORE -->
<div class="product-card">
  <h1 class="product-title">Nova Running Shoe</h1>
  <div class="price">
    <span class="amount">$89.99</span>       <!-- div.price > span.amount  : DIED -->
  </div>
  <button class="add-to-cart">Add to Cart</button>
</div>

<!-- AFTER -->
<div class="ProductCard_card__x7f2k">
  <h1 class="ProductCard_title__k9d3e">Nova Running Shoe</h1>
  <span data-price="89.99" data-currency="USD">$89.99</span>  <!-- div.price is gone -->
  <button aria-label="Add Nova Running Shoe to cart">Add to Cart</button>
</div>

Three selectors died here, and they died for three different reasons — which is the whole lesson of the diff. The div.price wrapper was removed entirely, so any descendant selector under it is unrecoverable at rung 1. The title class changed from a semantic name (product-title) to a CSS-modules hash (ProductCard_title__k9d3e), which means class-based matching is now fragile by design: the hash changes on every rebuild. The button kept its text but lost its stable class, gaining an aria-label instead.

Notice what the redesign gave you, though: data-price and data-currency attributes. Modern front-end frameworks emit data attributes because the framework’s own hydration layer needs them. Those are the most stable hooks on the page. The diff isn’t just a casualty report — it’s a map of the new, sometimes better, anchor points.

Rung 1: Rebuild CSS Selectors Against the New Markup

The cheapest fix is almost always rewriting your selectors. You already have the parsing code, the pipeline, the storage. Change a few strings, redeploy, done. But do it with some discipline about which selectors you choose, because rung 1 is also where you decide how soon the next redesign hurts you.

Here’s how I’d approach the product title on the new markup, from most fragile to most resilient:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

# OLD (dead): semantic class removed in redesign
# title = soup.select_one("h1.product-title")

# Candidate A — fragile: structural position, breaks on any layout tweak
title_a = soup.select_one("div[class*='ProductCard'] > h1:first-child")

# Candidate B — moderate: partial class match, survives hash changes
# but dies if the component is renamed
title_b = soup.select_one("h1[class*='title']")

# Candidate C — resilient: role + heading semantics, independent of styling
title_c = soup.select_one("[role='heading'][aria-level='1']") or soup.find("h1")

# Candidate D — most resilient: data attribute emitted for framework hydration
anchor = soup.select_one("[data-sku]")  # anchor on the hydration attribute
title_d = None
if anchor is not None:
    # real traversal: the heading lives inside the anchored product subtree
    title_d = anchor.find("h1") or anchor.select_one("[class*='title']")
# None-check fallback: if the anchor or the traversal came up empty,
# fall back to heading semantics (Candidate C's logic)
if title_d is None:
    title_d = soup.select_one("[role='heading'][aria-level='1']") or soup.find("h1")

My ranking, honestly: C and D over A and B, every time. Candidate A is what most people write under deadline pressure and it’s the reason this same outage recurs every few months. Candidate B looks clever but couples you to the site’s internal naming conventions, which are invisible to you and change without notice. Candidates C and D anchor on semantics — accessibility attributes and data attributes — that exist for the site’s own benefit, not yours. Sites rarely break their own accessibility tree or hydration layer to ship a visual redesign.

The trade-off matrix I use when choosing:

Strategy	Robustness	Readability	Maintenance cost
Class chains (div.price > span.amount)	Low — dies on any class rename	High — intent is obvious	High — breaks every redesign
data-* attributes ([data-price])	High — tied to app logic, not styling	High — self-documenting	Low — changes only when the app’s data model changes
ARIA roles ([role='heading'])	High — sites rarely break a11y trees	Medium — needs a comment or two	Low
Structural XPath (//div[2]/span[1])	Very low — breaks on any DOM reorder	Low — opaque to reviewers	Very high — silent wrong-field risk

That last row deserves emphasis: structural XPath doesn’t just break, it breaks silently into wrong data. When the DOM reorders, //div[2]/span[1] happily returns the shipping-policy text instead of the price, and your pipeline ingests garbage with no error. I covered this failure class in more depth in the piece on scraped data landing in wrong fields. If you inherit a scraper full of positional XPath, treat that as rung-0 debt.

Rung 1 fixes this outage. It does nothing about the next one. That’s what the ladder is for.

Rung 2: Fall Back to Structured Data Before You Parse HTML at All

Before you write a single new CSS selector, check whether the page is telling you the answer directly. Most e-commerce and content sites embed schema.org JSON-LD in a <script type="application/ld+json"> block, and that block is written for search engines, not browsers. It survives redesigns remarkably well, because breaking it costs the site its rich snippets.

This is the rung most people skip, and it’s the one I’d build on permanently. Check for it:

import json
from bs4 import BeautifulSoup

def extract_jsonld_product(html: str) -> dict | None:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(tag.string or "")
        except json.JSONDecodeError:
            continue
        # JSON-LD can be a single object or a graph (@graph)
        candidates = data.get("@graph", [data]) if isinstance(data, dict) else data
        for node in candidates:
            if isinstance(node, dict) and node.get("@type") in ("Product", ["Product"]):
                return node
    return None

def availability_code(availability) -> str | None:
    """Normalize schema.org availability to a short code, e.g. 'InStock'.

    Handles the three shapes you actually see in the wild: a plain URL
    string, a nested object like {'@type': 'https://schema.org/InStock'},
    and a missing or null value.
    """
    if isinstance(availability, str):
        return availability.rsplit("/", 1)[-1]
    if isinstance(availability, dict):
        nested = availability.get("@type") or availability.get("name")
        if isinstance(nested, str):
            return nested.rsplit("/", 1)[-1]
    return None

product = extract_jsonld_product(html)
if product:
    name = product.get("name")
    offer = product.get("offers") or {}
    price = offer.get("price") or offer.get("lowPrice")
    availability = availability_code(offer.get("availability"))  # e.g. "InStock"
    print(name, price, availability)

No selectors. No class names. No structural assumptions. The fields are labeled by contract — the schema.org vocabulary — rather than by styling decisions.

Here’s what that block typically looks like in the page head, with the mapping to the CSS-based fields you were scraping before:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Nova Running Shoe",            // <- was h1.product-title
  "sku": "NRS-123",                        // <- was div.product-card[data-id]
  "offers": {
    "@type": "Offer",
    "price": "89.99",                      // <- was div.price > span.amount
    "priceCurrency": "USD",                // <- was hardcoded in your parser
    "availability": "https://schema.org/InStock"  // <- was button.add-to-cart text
  }
}

Note what you gain beyond stability: priceCurrency was probably hardcoded in your old parser, and availability was inferred from button text like “Add to Cart” versus “Sold Out” — both fragile inferences that the structured data gives you explicitly.

The honest caveats. Not every page has JSON-LD, and not every site keeps it accurate — I’ve seen pages where the markup shows a sale price but the JSON-LD still carries the list price. Some sites render the block server-side only for crawler user-agents, so your fetch needs a browser-like one. And long-tail pages (old inventory, archived products) are the most likely to be missing the block entirely. So treat rung 2 as the preferred tier, not the only tier — which is exactly why the ladder exists.

Also check for the other embedded sources while you’re in there: microdata attributes (itemprop) and the __NEXT_DATA__-style JSON payloads that server-rendered frameworks inject. The hidden-JSON-endpoint question — whether to consume the site’s own data feed or parse the DOM — has its own trade-offs, which I covered in hidden JSON endpoints vs DOM parsing.

Rung 3: Prompt Extraction as the Last Resort — and Its Real Costs

When markup is gone and structured data is absent, the remaining option is handing the page text to an LLM with instructions on what to pull out. This works — it’s the whole premise of schema-driven extraction — but it’s the top rung for concrete reasons, not aesthetic ones.

Cost. An LLM call per page costs orders of magnitude more than a regex or a CSS selector. At 10,000 pages per day, per-token pricing turns your scraper from a nearly-free cron job into a line item someone will question in a budget review.

Latency. A selector resolves in microseconds. An LLM extraction adds seconds per page, which caps your throughput unless you parallelize aggressively.

Consistency. This is the underrated one. A selector returns the same string every time. An LLM can return "89.99", "$89.99", or 89.99 USD on different runs of the same page, and it can hallucinate a value when the field genuinely isn’t present. You must validate every response against a strict schema and treat validation failure as extraction failure — not as “close enough.”

Here’s the pattern, using the extraction endpoint of a scraping API so the fetch and the extraction happen in one call. FineData’s async scrape endpoint takes an extract_prompt alongside an extract_schema, which keeps the prompt and the validation contract in one place:

import requests

payload = {
    "url": "https://example.com",
    "formats": ["text"],
    "extract_prompt": (
        "Extract the product name, current price as a decimal number "
        "without currency symbols, currency code, and availability. "
        "If a field is not present on the page, return null for it. "
        "Never guess."
    ),
    "extract_schema": {
        "type": "object",
        "properties": {
            "name": {"type": ["string", "null"]},
            "price": {"type": ["number", "null"]},
            "currency": {"type": ["string", "null"]},
            "availability": {
                "type": ["string", "null"],
                "enum": ["in_stock", "out_of_stock"],
            },
        },
        "required": ["name", "price", "currency", "availability"],
    },
}

resp = requests.post(
    "https://api.finedata.ai/api/v1/async/scrape",
    headers={"Authorization": "Bearer fd_your_api_key"},
    json=payload,
    timeout=60,
)
job = resp.json()

Then poll or receive the result via webhook, and validate it on your side regardless of what the API returns:

from pydantic import BaseModel, ValidationError
from typing import Optional, Literal

class ProductRecord(BaseModel):
    name: Optional[str]
    price: Optional[float]
    currency: Optional[str]
    availability: Optional[Literal["in_stock", "out_of_stock"]]

try:
    record = ProductRecord.model_validate(job_result["extracted"])
except ValidationError as e:
    # Treat as extraction failure, route to a human-review queue
    log.warning("prompt extraction failed validation: %s", e)
    record = None

The Never guess line in the prompt and the null-allowed schema are doing the real reliability work here. The failure mode you’re defending against is a confident hallucinated price, which is worse than a null because nothing downstream notices it.

How the three rungs compare in practice:

	Rung 1: CSS rules	Rung 2: JSON-LD / schema	Rung 3: Prompt extraction
Lines of code	~5 per field	~20 one-time parser	~30 plus validation layer
Cost per 1,000 pages	~$0 (compute only)	~$0 (compute only)	Dollars to tens of dollars, model-dependent
Latency per page	Microseconds	Microseconds	Seconds
Failure modes	Silent nulls or wrong fields	Missing blocks; stale values	Hallucination; format drift
Maintenance burden	High — every redesign	Low — vocabulary is stable	Low code churn, but prompt tuning and validation upkeep

My opinion, stated plainly: prompt extraction is a recovery tool, not an architecture. Teams that make it rung one end up paying LLM prices for pages that had perfectly good JSON-LD sitting in the head, and they inherit a consistency problem they didn’t need. Use it as the safety net under the ladder, not as the floor.

Build the Escalation Ladder Into the Scraper, Not Your Memory

Knowing the ladder is worthless if the escalation happens by a human reading logs three days later. Wire the fallback into the extraction code itself, with per-tier logging so you can tell after the fact which rung served each field.

import logging
from bs4 import BeautifulSoup

log = logging.getLogger("extractor")

def prompt_extract_price(html):
    # full request/poll/validate flow from rung 3 above
    # (POST /api/v1/async/scrape with extract_prompt + extract_schema,
    #  poll the job or receive the webhook, validate with ProductRecord;
    #  returns None on validation failure)
    ...

def extract_product(html: str, url: str) -> dict:
    record = {"url": url, "served_by": {}}

    # Rung 2 first: structured data is the most reliable source when present
    jsonld = extract_jsonld_product(html)
    if jsonld:
        record["name"] = jsonld.get("name")
        record["price"] = (jsonld.get("offers") or {}).get("price")
        record["served_by"]["price"] = "rung2_jsonld"

    # Rung 1: CSS for anything JSON-LD didn't cover
    soup = BeautifulSoup(html, "html.parser")
    if record.get("price") is None:
        node = soup.select_one("[data-price]") or soup.select_one("span.amount")
        if node:
            record["price"] = node.get("data-price") or node.get_text(strip=True)
            record["served_by"]["price"] = "rung1_css"

    # Rung 3: prompt extraction as the final fallback, async so it
    # doesn't block the main pipeline
    if record.get("price") is None:
        record["price"] = prompt_extract_price(html)   # returns None on validation failure
        record["served_by"]["price"] = "rung3_prompt"

    if record.get("price") is None:
        record["served_by"]["price"] = "failed"

    log.info("extraction_tier url=%s field=price tier=%s",
             url, record["served_by"]["price"])
    return record

Two design points here. First, I try rung 2 before rung 1 in the built version, even though rung 1 is cheaper to fix by hand — because at runtime, structured data is both cheaper than an LLM call and more reliable than CSS. The ladder orders your repair effort; the runtime order should order by reliability. Second, the served_by map is the most valuable debugging artifact you’ll add this month. When the next redesign ships, your first question — “which fields degraded, and how far down the ladder did they fall?” — is answered by a group-by query instead of an investigation.

Then make degradation visible. A quiet fall from rung 1 to rung 3 is a cost leak and a quality risk, and it should page someone before the invoice does:

# Count pages whose price fell below rung 1, exported as a counter
from prometheus_client import Counter

EXTRACTION_TIER = Counter(
    "extraction_tier_total",
    "Extraction tier that served each field",
    ["field", "tier"],
)

def record_extraction_tier(record: dict) -> None:
    """The increment site — call from extract_product() just before `return record`."""
    for field, tier in record["served_by"].items():
        EXTRACTION_TIER.labels(field, tier).inc()

# Alert rule: more than 5% of pages on rung 2+ over 15 minutes
# - alert: ExtractionDegraded
#   expr: |
#     sum(rate(extraction_tier_total{tier=~"rung2.*|rung3.*|failed"}[15m]))
#       / sum(rate(extraction_tier_total[15m])) > 0.05
#   for: 10m

A 5% threshold with a 10-minute hold is a reasonable starting point — tight enough to catch a redesign within the hour, loose enough to ignore the occasional odd page. Tune it against your own baseline; if 30% of a site’s pages legitimately lack JSON-LD, exclude that site from the rung-2 term or you’ll train everyone to ignore the alert.

Redesign-Proofing Going Forward: Regression Tests and Golden Files

The ladder makes the next redesign survivable. Two cheap habits make it detectable within hours instead of weeks, and both run entirely on your side.

The first is golden-file tests. Save the raw HTML of a handful of representative pages — one per template type you scrape — into your repo, and assert your extraction against them in CI. When a selector change or a dependency upgrade breaks extraction, the test fails before deployment, not in production:

import pathlib
import pytest
from bs4 import BeautifulSoup

GOLDEN = pathlib.Path("golden_html/store.example.com")

@pytest.mark.parametrize("fixture_name", ["product_123.html", "product_456.html"])
def test_product_extraction(fixture_name):
    html = (GOLDEN / fixture_name).read_text()
    record = extract_product(html, url="https://store.example.com/")

    assert record["name"], f"name is null for {fixture_name}"
    assert record["price"] is not None, f"price is null for {fixture_name}"
    assert float(record["price"]) > 0, f"price not parseable for {fixture_name}"
    assert record["served_by"]["price"] in ("rung1_css", "rung2_jsonld"), (
        f"{fixture_name} degraded to {record['served_by']['price']}"
    )

That last assertion is the one that pays for itself. It doesn’t just check that extraction works — it checks that extraction works on the rung you intended. If a refactor quietly pushes every product onto prompt extraction, the test fails and you find out before the invoice does.

The second habit is canary scrapes. Run a small, fixed set of URLs — five to twenty pages covering each template — on an hourly cadence, separate from the main pipeline, with the tier metric from the previous section attached. The golden files catch your regressions; the canaries catch the site’s changes. You need both, because a redesign breaks your selectors without anything in your repo changing.

Here’s the detection math from a sample cadence, using a conservative assumption that a redesign lands at a uniformly random time between full runs:

Monitoring approach	Cadence	Expected time-to-detection	Worst case
Weekly full pipeline run	7 days	~3.5 days	7 days
Daily full run	1 day	~12 hours	24 hours
Hourly canary + daily full run	1 hour	~30 minutes	1 hour

The canary run costs a few dozen requests per hour — noise against any production budget — and it converts a multi-day silent outage into a sub-hour alert. That’s the best cost-to-benefit ratio in this entire post.

Wrap-Up

A redesign doesn’t have to be an incident. The playbook: confirm it’s the site and not you, diff the DOM to map exactly what died, then climb the ladder — resilient CSS selectors first, embedded JSON-LD as the tier you actually prefer, prompt extraction as the validated, logged last resort. Encode the fallback in code with per-field tier tracking, alert when extraction degrades past rung 1, and back it all with golden-file tests and hourly canaries.

The one-line version: selectors describe styling, structured data describes meaning, and meaning survives redesigns far better than styling. Build your extraction to prefer meaning whenever the page offers it.