Audit Field Fill Rates: Measure Extraction Quality Locally

Why Field Fill Rate Beats Row Count as a Scraped-Data Quality Metric

Row count is the laziest quality metric in web scraping. It tells you the scraper returned something for every URL. It says nothing about whether that something is usable. A scrape that returns 10,000 rows where 6,000 have a null price is worse than a scrape that returns 9,500 rows with clean prices — but the row-count dashboard shows the first one winning.

Here’s the comparison I keep coming back to. Two crawls of store.example.com product listings, same day, same target list:

Metric	Dataset A	Dataset B
Rows returned	10,000 (100%)	9,500 (95%)
title fill rate	100%	99.2%
price fill rate	40%	99.1%
sku fill rate	100%	98.8%
Usable rows (price present)	4,000	9,414
Cause	Selector drift on price node	500 pages returned 404

Dataset A “won” on row count and lost by a mile on everything that matters. The 404s in Dataset B are honest failures — you know exactly which 500 URLs to re-drive. The missing prices in Dataset A are silent corruption. They flow straight into your warehouse, and three weeks later someone asks why the price monitoring dashboard shows a 60% drop in coverage and nobody can reconstruct when it broke.

The naive check that hides this problem looks innocent enough:

import pandas as pd

df = pd.read_csv("store_example_products.csv")
expected = 10_000

if len(df) == expected:
    print("PASS: row count matches")
else:
    print(f"FAIL: got {len(df)}, expected {expected}")

Run that against Dataset A and it prints PASS. Run a fill-rate check against the same file and the story changes immediately:

price_fill = df["price"].notna().mean()
print(f"price fill rate: {price_fill:.1%}")   # price fill rate: 40.0%

That’s the whole argument. If you measure one thing about a scraped dataset before loading it, measure per-field fill rate. Everything else in this post builds on it.

Scaffolding the Audit Script: Loading Scraped Output and Declaring Expected Fields

The audit script should be standalone and dumb about where the data came from. It reads a local file — CSV or JSONL — and scores it against a manifest you declare up front. I keep the manifest in YAML next to the scraper code, because the manifest is the contract between extraction and ingestion, and it deserves to be versioned like code.

# audit_manifest.yaml
dataset: store_example_products
source: store.example.com
file: output/store_example_products.csv
fields:
  title:
    required: true
    min_fill_rate: 0.95
    max_stale_share: 0.30
  price:
    required: true
    min_fill_rate: 0.90
    max_stale_share: 0.30
    dtype: float
  sku:
    required: true
    min_fill_rate: 0.98
  availability:
    required: true
    min_fill_rate: 0.95
  image_url:
    required: false
    min_fill_rate: 0.80
checks:
  max_null_run: 50
  min_dataset_score: 85

Note the required: false on image_url. Not every field deserves the same bar. A scrape that drops 20% of thumbnail URLs is annoying; a scrape that drops 10% of prices is an incident. Setting per-field thresholds forces that conversation to happen explicitly instead of by accident.

Loading the file and printing a shape summary takes a few lines:

import pandas as pd
import yaml
import json

def load_scrape(path: str) -> pd.DataFrame:
    if path.endswith(".jsonl"):
        with open(path) as f:
            rows = [json.loads(line) for line in f if line.strip()]
        return pd.DataFrame(rows)
    return pd.read_csv(path)

with open("audit_manifest.yaml") as f:
    manifest = yaml.safe_load(f)

df = load_scrape(manifest["file"])
print(df.shape)
print(df.dtypes)
print(df.head(3))

The dtype summary is not decoration. A price column that loads as object instead of float64 means you have strings, currency symbols, or the literal string "null" in there — all of which will silently poison downstream math. Check dtypes early. It costs nothing.

One opinionated call: I load the raw scrape output, not a cleaned intermediate. If you audit after cleaning, you’re auditing your cleaning code, not your scraper. The whole point is to catch extraction failures before anything papers over them.

Computing Per-Field Fill Rate and Null Rate in One Pass

The core calculation is simple, but the details matter. A field can be missing in three distinct ways: a true NaN, an empty string, and a placeholder like "N/A" or "-" that your extractor emitted when the selector matched nothing. Count all three or you’ll overstate your fill rate.

import pandas as pd

PLACEHOLDERS = {"", "n/a", "N/A", "null", "NULL", "-", "none", "None"}

def field_fill_report(df: pd.DataFrame, fields: dict) -> pd.DataFrame:
    rows = []
    for name, cfg in fields.items():
        if name not in df.columns:
            rows.append({
                "field": name, "fill_rate": 0.0, "null_rate": 1.0,
                "empty_rate": 1.0, "n_rows": len(df), "status": "MISSING_COLUMN",
            })
            continue

        col = df[name]
        is_null = col.isna()
        is_empty = col.astype(str).str.strip().isin(PLACEHOLDERS) & ~is_null
        filled = ~is_null & ~is_empty

        fill_rate = filled.mean()
        threshold = cfg.get("min_fill_rate", 0.90)
        status = "PASS" if fill_rate >= threshold else "FAIL"

        rows.append({
            "field": name,
            "fill_rate": round(fill_rate, 4),
            "null_rate": round(is_null.mean(), 4),
            "empty_rate": round(is_empty.mean(), 4),
            "n_rows": len(df),
            "threshold": threshold,
            "status": status,
        })
    return pd.DataFrame(rows)

report = field_fill_report(df, manifest["fields"])
print(report.to_string(index=False))

The MISSING_COLUMN branch matters more than it looks. A template redesign that renames a key in the underlying JSON turns your whole column into NaN at load time — or removes it entirely — and the audit should fail loudly, not crash with a KeyError halfway through.

Terminal output for a run with a real problem:

   field  fill_rate  null_rate  empty_rate  n_rows  threshold status
    title     1.0000     0.0000      0.0000   10000       0.95   PASS
    price     0.6200     0.3100      0.0700   10000       0.90   FAIL
      sku     0.9950     0.0050      0.0000   10000       0.98   PASS
availability     0.9800     0.0100      0.0100   10000       0.95   PASS
  image_url     0.8300     0.1700      0.0000   10000       0.80   PASS

Price at 62% against a 90% floor. That’s a hard stop, and it’s visible in the first ten seconds of looking at the output — no warehouse load, no downstream confusion.

Detecting Null Clusters: When Missing Values Bunch Together

Fill rate tells you how much is missing. Null clustering tells you where. Those are different diagnoses. Randomly scattered nulls usually mean the target pages genuinely vary. A contiguous block of nulls almost always means something mechanical broke: a pagination boundary, a rate-limit window, a cookie that expired mid-crawl, or a selector change that only affects one template variant.

import pandas as pd

def longest_null_runs(df: pd.DataFrame, fields: dict) -> list[dict]:
    results = []
    for name in fields:
        if name not in df.columns:
            continue
        is_null = df[name].isna() | df[name].astype(str).str.strip().isin(PLACEHOLDERS)
        if not is_null.any():
            results.append({"field": name, "max_run": 0, "runs_over_limit": 0})
            continue

        # Label each contiguous block of nulls with a unique id
        block_id = (is_null != is_null.shift()).cumsum()
        null_blocks = is_null[is_null].groupby(block_id[is_null])

        runs = null_blocks.apply(lambda g: (g.index.min(), g.index.max(), len(g)))
        limit = manifest["checks"].get("max_null_run", 50)

        over = [r for r in runs if r[2] > limit]
        results.append({
            "field": name,
            "max_run": int(max(r[2] for r in runs)),
            "runs_over_limit": len(over),
            "worst_range": over[0][:2] if over else None,
        })
    return results

for r in longest_null_runs(df, manifest["fields"]):
    print(r)

The index ranges are the useful part. worst_range: (6400, 6599) on a paginated crawl of store.example.com maps directly to pages 128–132 of the crawl. You can go look at those pages, compare them to page 127, and usually see the template difference with your own eyes.

A concrete pattern I’ve hit repeatedly: a diff between two crawl runs of the same site shows description at 97% fill in run one and 78% in run two, with the entire deficit concentrated in a 200-row cluster starting at row 4,100. Row 4,100 was page 82. The site had shipped a redesigned product template for one category, and the description selector matched the old template only. Fill rate alone would have said “78%, mildly bad.” The cluster said “page 82 onward, one category, one template — go fix one selector.”

This is also where I’ll plant a flag people sometimes push back on: a dataset with a 200-row null cluster and 97% overall fill is, in my book, worse than one with 90% fill spread uniformly. Uniform missingness is often the data just being messy. Clusters are bugs, and bugs spread. Weight clustered failures harder than diffuse ones when you score.

Flagging Stale Values: Repeated Fields That Signal a Broken Selector

The sneakiest failure mode isn’t missing data — it’s wrong data that looks present. When a selector starts grabbing a static page element instead of the per-product value, every row gets filled and your fill rate reads a healthy 100%. The values are garbage.

The tell is value concentration. Real product titles on a store listing are near-unique. If one title occupies 71% of the column, the scraper is reading the page header, not the product name.

def staleness_report(df: pd.DataFrame, fields: dict, top_n: int = 3) -> list[dict]:
    results = []
    for name, cfg in fields.items():
        if name not in df.columns:
            continue
        counts = df[name].value_counts(dropna=True, normalize=True)
        if counts.empty:
            results.append({"field": name, "top_value": None, "share": 0.0, "status": "PASS"})
            continue

        top_value, share = counts.index[0], counts.iloc[0]
        limit = cfg.get("max_stale_share", 0.30)
        results.append({
            "field": name,
            "top_value": str(top_value)[:60],
            "share": round(float(share), 4),
            "limit": limit,
            "status": "FAIL" if share > limit else "PASS",
            "top_values": [
                {"value": str(v)[:60], "share": round(float(s), 4)}
                for v, s in counts.head(top_n).items()
            ],
        })
    return results

for r in staleness_report(df, manifest["fields"]):
    print(r)

The before/after that convinced me this check belongs in every audit: a store.example.com scrape where the title column came back at 100% fill rate and 71% of the values read Product Details. A redesign had moved the real title into a nested component the old selector didn’t reach, while the old selector now matched the page’s static heading. Fill-rate-only auditing shipped that dataset for two days before a human noticed. The staleness check catches it in one pass — share: 0.71 against a 0.30 limit, status FAIL.

One caveat: this check needs field-level tuning. availability legitimately has two or three distinct values, so a 60% share of in_stock is fine. That’s why max_stale_share lives in the per-field manifest rather than in a global config — high-cardinality fields like title, sku, and price get tight limits; enum-like fields get loose ones or none.

Scoring the Whole Dataset: A Single Pass/Fail Audit Report

Now combine the three checks into one score a machine can act on. The design goal: a single exit code for CI and pre-ingestion gating, plus a JSON artifact you can keep for trend analysis.

My scoring weights, after running this on enough crawls to have opinions: fill rate counts most (50 points), staleness next (30), clusters last (20). A field can pass fill rate and still fail staleness, and I want that to hurt the score enough to block the load.

import json
import sys

def score_dataset(fill_report, cluster_report, stale_report, manifest) -> dict:
    fields = manifest["fields"]
    per_field = {}

    for _, row in fill_report.iterrows():
        name = row["field"]
        fill_score = min(row["fill_rate"] / row.get("threshold", 0.90), 1.0) * 50

        stale = next((s for s in stale_report if s["field"] == name), {})
        stale_share = stale.get("share", 0.0)
        stale_ok = stale_share <= fields.get(name, {}).get("max_stale_share", 1.0)
        stale_score = (30 if stale_ok else 30 * (1 - stale_share)) if stale else 30

        cluster = next((c for c in cluster_report if c["field"] == name), {})
        max_run = cluster.get("max_run", 0)
        limit = manifest["checks"].get("max_null_run", 50)
        cluster_score = 20 if max_run <= limit else max(0, 20 - (max_run - limit) / 20)

        hard_fail = row["status"] == "FAIL" or not stale_ok
        per_field[name] = {
            "fill_rate": row["fill_rate"],
            "stale_share": stale_share,
            "max_null_run": max_run,
            "field_score": round(fill_score + stale_score + cluster_score, 1),
            "hard_fail": hard_fail,
        }

    score = round(
        sum(v["field_score"] for v in per_field.values()) / len(per_field), 1
    )
    threshold = manifest["checks"].get("min_dataset_score", 85)
    any_hard_fail = any(v["hard_fail"] for v in per_field.values())
    verdict = "PASS" if score >= threshold and not any_hard_fail else "FAIL"

    return {
        "dataset": manifest["dataset"],
        "n_rows": int(fill_report["n_rows"].max()),
        "score": score,
        "threshold": threshold,
        "verdict": verdict,
        "fields": per_field,
    }

report = score_dataset(field_fill_report(df, fields), longest_null_runs(df, fields),
                       staleness_report(df, fields), manifest)

with open("audit_report.json", "w") as f:
    json.dump(report, f, indent=2)

print(f"Audit verdict: {report['verdict']} (score {report['score']})")
sys.exit(0 if report["verdict"] == "PASS" else 1)

The hard_fail flag is deliberate. A weighted average can average its way to a passing score while one critical field is broken — 40% price fill drags the score down but a strong everything-else can still clear 85. I don’t want that dataset loading. Any required field failing its own threshold vetoes the run regardless of the aggregate.

The artifact for a failing run of store.example.com products:

{
  "dataset": "store_example_products",
  "n_rows": 10000,
  "score": 61.4,
  "threshold": 85,
  "verdict": "FAIL",
  "fields": {
    "title": {"fill_rate": 1.0, "stale_share": 0.71, "max_null_run": 0, "field_score": 51.3, "hard_fail": true},
    "price": {"fill_rate": 0.62, "stale_share": 0.0, "max_null_run": 210, "field_score": 34.4, "hard_fail": true},
    "sku": {"fill_rate": 0.995, "stale_share": 0.0, "max_null_run": 3, "field_score": 99.8, "hard_fail": false},
    "availability": {"fill_rate": 0.98, "stale_share": 0.54, "max_null_run": 0, "field_score": 92.0, "hard_fail": false},
    "image_url": {"fill_rate": 0.83, "stale_share": 0.0, "max_null_run": 45, "field_score": 88.7, "hard_fail": false}
  }
}

Read that report top to bottom and you know the story before opening a single page: title selector grabbing a static element, price broken past a pagination boundary, everything else healthy. Five seconds of diagnosis from one file.

Wiring the Audit into Your Pipeline: Run It Before the Warehouse Load

An audit that nobody runs automatically is documentation, not a control. Put it between scraping and loading, and make the load step structurally unable to run when the audit fails.

A Makefile version:

SCRAPE_TARGETS := urls/store_example_products.txt
OUTPUT := output/store_example_products.csv

scrape:
	python scraper.py --urls $(SCRAPE_TARGETS) --out $(OUTPUT)

audit: scrape
	python audit.py --manifest audit_manifest.yaml
	@echo "Audit passed, writing report"

load: audit
	python load_to_warehouse.py --input $(OUTPUT) --table products
	@echo "Load complete"

Because load depends on audit, and audit.py exits non-zero on failure, make load stops before touching the warehouse. No if statements, no discipline required — the dependency chain does the enforcing.

If your scrape step uses an async batch API, the same chain applies: the scraper polls GET /api/v1/async/batch/{batch_id} with include_results=true, writes the JSONL locally, and the audit runs on that file. The audit never talks to the network, which is exactly what you want — a quality gate that can’t fail because a remote endpoint hiccuped.

For CI, run the audit against a small committed fixture so a selector regression gets caught at commit time instead of at 3 a.m. during the scheduled crawl:

name: audit-fixture
on: [push]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pandas pyyaml
      - run: python audit.py --manifest fixtures/store_example_manifest.yaml

The fixture is a 50-row JSONL snapshot of a known-good scrape, plus a deliberately broken variant in a separate test that asserts the audit fails on it. Testing your quality gate is not optional — an audit script with a bug in it that always passes is worse than no audit script, because it manufactures false confidence.

Extending the Audit: Cross-Field Consistency and Trend Tracking Over Time

Two upgrades once the basics are running.

First, cross-field rules. Individual fields can pass every threshold while combinations are impossible. A product marked in_stock with a null price is broken data no per-column check will ever see:

def cross_field_violations(df: pd.DataFrame) -> dict[str, int]:
    price_missing = df["price"].isna() | df["price"].astype(str).str.strip().isin(PLACEHOLDERS)
    violations = {
        "in_stock_null_price": int(
            ((df["availability"] == "in_stock") & price_missing).sum()
        ),
        "title_without_sku": int(
            (df["title"].notna() & df["sku"].isna()).sum()
        ),
    }
    return violations

print(cross_field_violations(df))
# {'in_stock_null_price': 233, 'title_without_sku': 12}

Start with two or three rules derived from how the data is actually consumed downstream. If the pricing team filters on in_stock, the in_stock_null_price rule is your highest-value check. Don’t write twenty speculative rules — you’ll spend more time triaging false positives than catching real failures.

Second, trend tracking. A single audit answers “is this run good?” A trend log answers “is quality decaying?” — which is often the more important question, because slow decay never trips a threshold until it’s already bad. Append the score from every run to a small SQLite table:

import sqlite3
from datetime import date

conn = sqlite3.connect("audit_trend.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS audit_runs (
        run_date TEXT, dataset TEXT, score REAL, verdict TEXT
    )
""")
conn.execute(
    "INSERT INTO audit_runs VALUES (?, ?, ?, ?)",
    (date.today().isoformat(), report["dataset"], report["score"], report["verdict"]),
)
conn.commit()

The pattern that justifies the whole exercise looks like this when you query the log:

run_date    score
----------  -----
week 1      98.2
week 1+2d   97.9
week 1+4d   96.1
week 2      93.0
week 2+3d   88.4
week 2+5d   84.1

No single day-to-day drop looks alarming. Over two weeks, price fill slid from 98% to 84% — the site was progressively migrating product pages to a new template, a few hundred per day. Every individual run passed its threshold until one didn’t. The trend log surfaces the slope weeks before the threshold fires, which is the difference between “schedule a selector fix this sprint” and “explain to stakeholders why price coverage collapsed.”

If you’re re-scraping on a schedule anyway, pair this with a change-detection strategy so you’re not paying to re-fetch pages that haven’t changed — I covered the trade-offs in Re-Scrape vs Change Detection: Choosing a Refresh Strategy. And when the audit does fail, re-drive only the failed URLs rather than the whole crawl; the mechanics for partial batch recovery are in Re-Drive Only the Failures: Handling Partial Batch Results.

Wrap-Up

The uncomfortable truth about scraped data is that nobody notices bad rows until they’ve already been consumed. Row counts hide selector drift. Fill rate exposes it. Null clusters localize it. Staleness checks catch the failures that fill rate can’t see at all — wrong values wearing the costume of present values.

The full audit is maybe 150 lines of Python with no dependencies beyond pandas and PyYAML. It runs in seconds on files of a hundred thousand rows. Wire it in as a gate, commit a fixture, keep the trend log, and the next time a target site ships a redesign, you’ll find out from a failing CI job instead of from a confused analyst.

Start small: manifest, fill rates, exit code. Add clusters and staleness once the basics are boring. The first silent failure it catches pays for the whole build.