Measure Duplicate Pages in Your Scraped Dataset

Why Exact MD5 Hashing Misses Boilerplate Near-Duplicates

You scraped 40,000 pages, deduplicated them by MD5, and the pipeline reports a clean 2.8% duplicate rate. Then someone opens the dataset and finds the same product page eleven times with different query strings. The MD5 pass caught none of them, because byte-level hashing only flags pages that are bit-for-bit identical. Real scraped duplication is messier.

Here is what that looks like in practice. Three fetches of the same content:

URL	Visible text	MD5 of raw HTML	Notes
https://example.com/page-a	”Product A costs $19.99…”	9f2c...	Canonical
https://store.example.com/page-a	”Product A costs $19.99…”	b71e...	Rotated A/B banner in <header>
https://example.com/page-a?utm=ref	”Product A costs $19.99…”	e04d...	CSRF token + cache-buster differ

All three render identically to a human. All three hash differently. A timestamp in the footer, a per-request CSRF token, a randomly chosen promotional banner — any of these breaks an exact hash while leaving the content untouched. If your dedup step is md5(html), you are measuring “byte-identical rate,” not duplication.

The gap is not small. Run this on a 1,000-page sample from a templated site and count what each method reports:

import hashlib, json

exact = set()
near_misses = []

with open("crawl_sample.jsonl") as f:
    records = [json.loads(line) for line in f]

for r in records:
    exact.add(hashlib.md5(r["html"].encode()).hexdigest())

exact_ratio = 1 - len(exact) / len(records)
print(f"Exact duplicates (MD5):  {exact_ratio:.1%}")
# Manual spot-check of 100 pages flagged 174 near-duplicate pairs
print(f"Near duplicates (manual): 17.4%")

On a crawl where MD5 reports 2.8%, a manual pass over the same sample routinely surfaces 15-18% near-duplication. That difference is the difference between a dataset you can trust for price analytics and one that silently double-counts every product that got mirrored onto a regional subdomain.

Before you fix it, you need to measure it. That is what this post builds: a local simhash checker that runs over your existing scrape dump, flags near-duplicate pairs, and prints a duplication ratio you can actually defend in a review. If you store raw HTML per URL — and you should, for reasons covered in Parse at Fetch Time vs Store Raw HTML — you already have everything the checker needs.

How Simhash Collapses a Page into a 64-Bit Fingerprint

Simhash is a locality-sensitive hash. Regular hashes amplify tiny input differences; simhash absorbs them. Two pages that differ by a timestamp produce fingerprints that differ by one or two bits, not completely. That property is the whole point.

The algorithm has three steps:

1. Shingle the text into overlapping n-gram windows.
2. Hash each shingle to a 64-bit integer.
3. Vote per bit position: for each of the 64 bit columns, add the shingle’s hash weight if the bit is 1, subtract if 0. The final fingerprint bit is 1 if the column sum is positive, 0 otherwise.

Here is a compact implementation you can run on a paragraph:

import hashlib

def shingles(text, size=5):
    words = text.split()
    return [" ".join(words[i:i + size]) for i in range(len(words) - size + 1)]

def hash64(token):
    return int.from_bytes(
        hashlib.md5(token.encode("utf-8")).digest()[:8], "big"
    )

def simhash(text, size=5):
    votes = [0] * 64
    for sh in shingles(text, size):
        h = hash64(sh)
        for i in range(64):
            votes[i] += 1 if (h >> i) & 1 else -1
    fingerprint = 0
    for i in range(64):
        if votes[i] > 0:
            fingerprint |= (1 << i)
    return fingerprint

sample = "Product A costs 19.99 and ships in two business days from our warehouse"
print(f"{simhash(sample):016x}")

Why does this tolerate noise? A single changed token (say, a timestamp word) only affects the 5-word shingles that contain it. With 60 shingles in the paragraph, five flipped shingle hashes shift five votes in each bit column. The remaining 55 shingles outvote them, and the fingerprint barely moves — or moves by a couple of bits at most.

Similarity between two pages is then just Hamming distance: count the bit positions where the two fingerprints differ.

def hamming(a, b):
    return bin(a ^ b).count("1")

about_v1 = simhash("Example Corp builds data pipelines for engineering teams in Berlin")
about_v2 = simhash("Example Corp builds data pipelines for engineering teams in Munich")

d = hamming(about_v1, about_v2)
print(f"Hamming distance: {d}")  # e.g. Hamming distance: 2

Two bits apart. A rewritten paragraph or a genuinely different page lands 20+ bits apart. The gap between “same page, trivially edited” and “different page” is wide enough that a threshold of 3 bits separates them cleanly for most content sites.

One caveat worth internalizing: simhash is not magic similarity detection. It measures shingle overlap. Two pages that quote the same long boilerplate paragraph will drift close even if everything else differs. You will see how to handle that in the normalization and auditing sections — it is the single biggest source of false positives, and pretending it does not exist is how teams end up deleting unique product pages.

Configuring the Checker: Shingle Size, Fingerprint Bits, and Distance Threshold

Three parameters control everything. Set them before you run anything at scale, because they determine whether your duplication ratio is honest or noise.

• Shingle size — how many words per window. Small shingles (3) make the fingerprint sensitive to local edits; large shingles (8) require substantial overlap to move the fingerprint at all.
• Fingerprint bits — 64 is the standard. Going to 128 reduces accidental collisions but doubles your index size for marginal benefit. Stay at 64.
• Hamming threshold — pairs at or below this distance count as duplicates. 3 bits is a common default; 1 is paranoid; 6 starts swallowing cousins.

Store the config next to the checker so results are reproducible:

# dupcheck.yaml
shingle_size: 5
fingerprint_bits: 64
hamming_threshold: 3
bands: 4          # for candidate pair generation, see below

How the choices interact on a real templated site — store.example.com product pages, which share a header, spec table skeleton, and footer:

Shingle size	Threshold	Expected behavior on templated product pages
3	1 bit	Flags only re-fetches and query-string mirrors. Misses pages whose boilerplate rotated.
3	3 bits	Catches mirrors plus lightly edited re-saves. Some false pairs on spec-table-heavy pages.
5	3 bits	Best default. Mirrors, translations of the same SKU page, and stale re-crawls all cluster.
5	6 bits	Starts grouping sibling products that share long spec tables. Risky.
8	3 bits	Insensitive to short edits; only near-identical full pages match. Undercounts.
8	6 bits	Unpredictable — large shingles plus loose threshold produce inconsistent clusters.

My recommendation: start at 5-word shingles with a 3-bit threshold, run the audit pass described later, and only then adjust. Tuning parameters after looking at flagged pairs is fine; tuning them to hit a target duplication ratio is not. I have seen teams lower the threshold until the dedup number looked acceptable in a stakeholder deck. That is not measurement, that is arithmetic theater.

Normalizing Scraped HTML: Stripping Nav, Footer, and Script Noise

Raw HTML is the wrong input for simhash. Navigation bars, footers, and script blocks are identical across every page on a domain — leave them in and every page on example.com lands within a few bits of every other. The similarity signal drowns in boilerplate.

Strip the structural chrome first, then extract text:

from bs4 import BeautifulSoup

STRIP_TAGS = ["nav", "footer", "script", "style", "form", "header", "aside"]

def normalize(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(STRIP_TAGS):
        tag.decompose()
    text = " ".join(soup.stripped_strings)
    # collapse whitespace so tokenization is stable
    return " ".join(text.split())

The effect is dramatic. On a representative example.com product page:

Stage	Extracted text length	Simhash (hex)
Raw HTML, all tags	14,200 chars	c3a91f02be47d108
After stripping nav/footer/script/form	1,850 chars	5e02aa71c9df3b40

The fingerprints are unrelated — which is correct, because they describe different text. The normalized fingerprint now reflects the content of the page, so two genuinely different products on the same template land far apart, while a re-fetch of the same product lands close.

Two opinions here, because normalization choices are where reasonable people disagree:

First, I strip <header> and <aside> aggressively. Some argue page headers contain meaningful content (product category breadcrumbs). In my experience breadcrumbs add more cross-page similarity than signal — every product in a category shares them. If your analysis needs breadcrumbs, extract them as a separate structured field rather than leaving them in the text you fingerprint.

Second, do not normalize URLs, prices, or dates out of the text. If a price changed from $19.99 to $24.99, the pages should hash differently. Your dedup pass runs on the current crawl; a price change means the old snapshot is stale, not duplicate — that is a refresh-strategy question, covered well in Re-Scrape vs Change Detection, not a dedup question.

If you are fetching pages through an API rather than parsing stored HTML, request only_main_content: true at scrape time to get most of this for free — but still normalize defensively, since the definition of “main content” varies by site.

Batch-Computing Fingerprints Across Your Crawl Records

Now the full pass. Assume your crawl dump is JSONL, one record per line, each with a url and html (or normalized text). Process it once, emit (url, fingerprint) pairs, and store them — you will re-run comparisons far more often than re-computation.

import json

def compute_fingerprints(input_path, output_path, cfg):
    with open(input_path) as fin, open(output_path, "w") as fout:
        for line in fin:
            rec = json.loads(line)
            text = normalize(rec["html"]) if "html" in rec else rec["text"]
            if len(text.split()) < cfg["shingle_size"]:
                continue  # too short to fingerprint meaningfully
            fp = simhash(text, cfg["shingle_size"])
            fout.write(json.dumps({
                "url": rec["url"],
                "fp": f"{fp:016x}",
                "len": len(text),
            }) + "\n")

compute_fingerprints(
    "crawl.jsonl", "fingerprints.jsonl",
    {"shingle_size": 5},
)

Note the skip for short pages. A page with three words of text (“Access denied”, “Loading…”) produces a degenerate fingerprint that matches everything else that short. Dropping sub-shingle pages is not data loss; those records are failures you should count separately. If your dump is full of them, your scrape has a block-page problem — read Your 200 OK Responses Are Actually Block Pages before worrying about dedup.

First five records from a mixed crawl of https://example.com and https://store.example.com/products/42 style URLs:

{"url": "https://example.com", "fp": "5e02aa71c9df3b40", "len": 1832}
{"url": "https://example.com/about", "fp": "a19c4d03e7f2b866", "len": 2401}
{"url": "https://store.example.com/products/42", "fp": "c07be15d92aa3f4e", "len": 1710}
{"url": "https://store.example.com/products/42?utm=affiliate", "fp": "c07be15d92aa3f4e", "len": 1710}
{"url": "https://example.com/about?print=1", "fp": "a19c4d03e7f2b866", "len": 2401}

The mirror URLs already show identical fingerprints. That is the easy 2.8%. The near-duplicates hiding one or two bits apart are next.

Finding Candidate Pairs with Hamming Distance and Banding Buckets

You have 40,000 fingerprints. Comparing every pair is 40,000 × 39,999 / 2 ≈ 800 million Hamming computations. In Python that is minutes to hours of wasted time, and 99.99% of the comparisons are between unrelated pages.

Banding fixes this. Split each 64-bit fingerprint into b blocks. If two fingerprints differ in at most t bits, then at least b − t of their blocks are identical. So: bucket every fingerprint by each of its blocks, and only pages sharing at least one block value become candidate pairs. Verify each candidate with exact Hamming distance afterward.

from collections import defaultdict

def candidate_pairs(fps, bands=4, threshold=3):
    """fps: list of (index, fingerprint_int). Yields candidate index pairs."""
    block_bits = 64 // bands
    buckets = defaultdict(list)

    for idx, fp in fps:
        for b in range(bands):
            block = (fp >> (b * block_bits)) & ((1 << block_bits) - 1)
            buckets[(b, block)].append(idx)

    seen = set()
    for members in buckets.values():
        if len(members) < 2:
            continue
        for i in range(len(members)):
            for j in range(i + 1, len(members)):
                pair = (members[i], members[j])
                if pair not in seen:
                    seen.add(pair)
                    yield pair

def verify_pairs(fps, candidates, threshold=3):
    for i, j in candidates:
        d = bin(fps[i][1] ^ fps[j][1]).count("1")
        if d <= threshold:
            yield (i, j, d)

With 4 bands and a threshold of 3, any true duplicate pair is guaranteed to share at least one 16-bit block, so banding loses no recall at that threshold. The only cost is candidate pairs that fail verification — cheap XORs.

Measured on local hardware, same dataset, same Python process:

Dataset	Brute-force pairs	Banded (4 blocks)	Runtime
10k pages, store.example.com	~50M	~180k candidates	96s → 1.4s
50k pages, store.example.com	~1.25B	~950k candidates	~40min → 7.8s

The speedup compounds because bucket membership grows sub-linearly for real data — most 16-bit block values are unique across unrelated pages. If your buckets get huge (a single block value with 5,000 members), that is itself a diagnostic: you have thousands of pages sharing a template down to the shingle level, and the audit pass in the next section just became mandatory.

Computing and Interpreting Your True Duplication Ratio

Flagged pairs are not yet a number. Pairs chain — A matches B, B matches C, but A and C sit 5 bits apart. Union-find clustering turns the pair list into groups, and each group contributes exactly one keepable representative:

def duplication_report(urls, fps, pairs, threshold_clusters=True):
    parent = list(range(len(urls)))

    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    def union(a, b):
        ra, rb = find(a), find(b)
        if ra != rb:
            parent[rb] = ra

    for i, j, _ in pairs:
        union(i, j)

    clusters = defaultdict(list)
    for i in range(len(urls)):
        clusters[find(i)].append(i)

    total = len(urls)
    dup_groups = [m for m in clusters.values() if len(m) > 1]
    redundant = sum(len(m) - 1 for m in dup_groups)

    print(f"Total URLs:            {total}")
    print(f"Duplicate clusters:   {len(dup_groups)}")
    print(f"Redundant pages:       {redundant} ({redundant/total:.1%})")
    print(f"Unique pages:          {total - redundant} ({(total-redundant)/total:.1%})")

Sample report from a 40,000-URL crawl of store.example.com plus its regional mirrors:

Total URLs:            40000
Exact duplicates:      1240  (3.1%)
Near duplicates:       5880  (14.7%)
Unique pages:          32880 (82.2%)
Duplicate clusters:    2210  (largest cluster: 11 URLs)
Redundant pages:       7120  (17.8%)

Read that as three separate signals:

• Exact duplicates (3.1%) are a crawl-hygiene problem. Your URL frontier is admitting ?utm= mirrors and HTTP/HTTPS twins. Fix the frontier, and this number drops on the next run. See Sitemap Crawling vs Link Discovery for canonicalization strategies at discovery time.
• Near duplicates (14.7%) are the real finding. These are stale re-crawls, regional mirrors, and pagination artifacts that MD5 never saw.
• Largest cluster size is your canary. A cluster of 11 is a mirrored product. A cluster of 400 is a template false-positive problem, and the next section applies.

One number gates downstream analytics: 82.2% unique. Any per-page average, price index, or trend line computed on the raw 40,000 URLs is inflated by the 17.8% redundant mass. If product 42 appears eleven times, its price gets eleven votes. Deduplicate first, compute second — always in that order.

Auditing Flags: Separating Templated Pages from Real Duplicates

Do not feed the flagged pairs straight into a DELETE statement. Simhash measures shingle overlap, and templates manufacture shingle overlap between genuinely different pages. Before dropping anything, sample 20-30 flagged pairs and look at them.

Here is a typical false positive from the audit. Two flagged store.example.com product pages, 90% shared template text:

--- products/42 (wireless mouse) ---        --- products/43 (wireless keyboard) ---
[shared: 1,340 chars of spec table,        [shared: identical spec table skeleton,
 shipping policy, category blurb]            shipping policy, category blurb]
diff: "Ergonomic wireless mouse with        diff: "Low-profile wireless keyboard with
       silent-click switches, 2.4GHz..."           membrane keys, 2.4GHz receiver..."

Hamming distance: 2 bits. The checker is technically correct — these pages are 90% identical text. They are also different products, and deleting one corrupts your catalog. The failure is not the algorithm; it is that the unique signal (the product description) is a small fraction of total text. That is a content-density problem, and you can quantify it with the approach in Measure Content Density.

Turn the audit into a decision table and apply it mechanically:

Flag category (from audit sample)	Example	Action
Query-string / mirror duplicates	Same product, ?utm= and canonical URL	Drop one. Fix frontier canonicalization.
Stale re-crawl of a changed page	Old price snapshot vs current	Keep newest by crawl timestamp. Feed the change into refresh strategy.
Regional mirror	store.example.com vs de.store.example.com same SKU	Keep both if you track per-region pricing; tag as same product ID.
Templated siblings, different products	Mouse vs keyboard, 2 bits apart	Keep both. Tighten normalization (strip spec table) or raise shingle size to 8.
Everything within 1 bit of everything	Whole domain clustering together	Normalization is broken — you are fingerprinting boilerplate. Fix before proceeding.
Short/degenerate pages in clusters	”Access denied” pages matching each other	Not duplicates. These are scrape failures; count them as a success-rate problem.

The audit pass costs twenty minutes and catches the failure mode that costs weeks: silently deleting unique pages because they share a template. It also tells you which parameter to retune, if any. Mouse-vs-keyboard false positives mean the spec table needs stripping; mirror-only clusters mean your config is already right.

Wrap-Up

Exact hashing measures byte-identity, which is the wrong metric for scraped data. Simhash measures content similarity, survives CSRF tokens and rotated banners, and with banding it scales to tens of thousands of pages in seconds. The pipeline you just built has four moving parts — normalize, fingerprint, band-and-verify, cluster — and each one is debuggable in isolation.

Three things to carry forward:

1. Report three numbers, not one. Exact duplicates, near duplicates, and unique pages tell different stories about crawl hygiene, staleness, and template noise respectively.
2. Audit before deleting. Twenty minutes of spot-checking prevents the template false-positive that deletes real products.
3. Gate analytics on the unique count. Every average computed over a deduplicated set is a different number than the same average over raw URLs, and only one of them is true.

Gotchas worth repeating: skip pages shorter than your shingle size or they will match everything; do not normalize prices and dates out of the text or you will merge stale and current snapshots; and if a single banding bucket holds thousands of members, your normalization is fingerprinting boilerplate, not content.

Next steps: wire the duplication report into your pipeline as a post-crawl quality gate, pair it with Audit Field Fill Rates for a fuller extraction-quality picture, and re-run the ratio after every frontier change so you can see whether your crawl hygiene is actually improving.