Re-Scrape vs Change Detection: Choosing a Refresh Strategy

The Refresh Problem: Why Re-Fetching Everything Costs You

Every scraping pipeline eventually hits the same question: how often should you re-fetch pages you already have? The default answer is a scheduled full re-scrape — simple, always correct, quietly expensive. The alternative is change detection: use signals the target already emits (ETags, sitemap lastmod stamps, content hashes) so you only spend requests on pages that actually moved.

This is a comparison of four refresh strategies, all measured against one model target: a storefront at store.example.com with 10,000 product pages averaging 48 KB of HTML each, and a politeness ceiling of 3 requests per second. Every number below traces back to that model, so you can rerun the same measurement against your own target before committing to an architecture.

The Full Re-Scrape Baseline: Measuring What a Blanket Crawl of store.example.com Actually Costs

Don’t pick a refresh strategy on vibes. Pick it after timing the dumb one. Here’s the entire baseline crawler:

import time
import requests

BASE = "https://store.example.com"
urls = [f"{BASE}/products/{i}" for i in range(10_000)]  # from your URL inventory

RATE = 3  # req/s ceiling
session = requests.Session()
bytes_down = 0
t0 = time.monotonic()

for url in urls:
    resp = session.get(url, timeout=30)
    resp.raise_for_status()
    bytes_down += len(resp.content)
    time.sleep(1 / RATE)

elapsed = time.monotonic() - t0
print(f"requests:   {len(urls)}")
print(f"downloaded: {bytes_down / 1e6:.0f} MB")
print(f"duration:   {elapsed / 60:.1f} min")

One run produces the numbers that justify or kill every optimization you’re about to consider:

Metric	Value
Pages	10,000
Requests issued	10,000
Bytes downloaded (uncompressed)	~480 MB
Wall clock at 3 req/s	~55 min
Pages whose content actually changed	12

That last row is the indictment. You transferred 480 MB and burned 55 minutes to discover 12 changed pages. The other 9,988 fetches returned bytes you already had on disk.

Two things worth noticing. First, len(resp.content) counts post-decompression bytes; with gzip the wire transfer is smaller, but the ratio of waste is identical. Second, the 55-minute duration is pure rate-limit arithmetic — 10,000 requests simply cannot complete faster at 3 req/s, no matter how clever your HTTP client is. The only lever that shortens the window is issuing fewer requests. Whether blocking loops or job queues suit your crawl is a separate question — sync vs async scraping — but neither touches the request count, which is the actual constraint here.

Conditional GET with ETags and If-None-Match: Skipping Unchanged Pages at the HTTP Layer

HTTP solved this problem decades ago and almost nobody uses it in scrapers. The server hands you a validator — an ETag — and if you replay it on the next request, the server can answer “nothing changed” with a bodyless 304.

Capture the validator first:

$ curl -sI https://store.example.com/products/42
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
ETag: "a8f3e-1d2b9c-64f0a1"
Last-Modified: <http-date>

Then replay it:

$ curl -s -o /dev/null -w "%{http_code}\n" \
    -H 'If-None-Match: "a8f3e-1d2b9c-64f0a1"' \
    https://store.example.com/products/42
304

No body came back. Your cached copy is still valid. In Python, persist the validators between runs and branch on the status code:

etags = load_etags()  # {url: '"a8f3e-1d2b9c-64f0a1"', ...} persisted to disk

def fetch_if_changed(url, session):
    headers = {"If-None-Match": etags[url]} if url in etags else {}
    resp = session.get(url, headers=headers, timeout=30)
    if resp.status_code == 304:
        return None                    # unchanged: ~0.5 KB of headers, no body
    etags[url] = resp.headers.get("ETag", "")
    return resp.text                   # 200: new body, refresh the validator

For servers that don’t issue ETags, If-Modified-Since with the stored Last-Modified date does the same job with coarser granularity.

Run the numbers: 9,949 responses of headers-only plus 51 full bodies works out to roughly 10 MB instead of 480 MB. That’s a ~98% bandwidth cut for maybe twenty lines of code.

Here’s the part that disappoints people: request count doesn’t move. It’s still 10,000 requests, the same rate-limit budget, the same origin load, the same 55-minute crawl window. Conditional GET cuts bytes, not requests. If your pain is egress cost or bandwidth, it’s a great fix. If your pain is origin pressure, crawl duration, or staying under a WAF’s radar, it does almost nothing.

There are also trust issues. Some CDNs strip or rewrite ETags. Some origins emit weak validators (W/"...") that miss fine-grained changes. Worst case, a site returns the same ETag for modified content — a false-negative generator you’ll meet again in the checklist at the end.

Sitemap lastmod as a Cheap Pre-Filter Before You Spend Any Bandwidth

The sitemap is a single request that describes the change state of the entire site. When it’s maintained honestly, it’s the highest ratio of value to code in this entire article: one request can eliminate 9,660 others.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <!-- lastmod shown as an offset from "now" (T); real files carry full ISO-8601 stamps -->
  <url>
    <loc>https://store.example.com/products/sapphire-mug</loc>
    <lastmod>T-00:14</lastmod>  <!-- edited 14 minutes ago -->
  </url>
  <url>
    <loc>https://store.example.com/products/cedar-plank</loc>
    <lastmod>T-2d</lastmod>
  </url>
  <url>
    <loc>https://store.example.com/products/linen-runner</loc>
    <lastmod>T-34d</lastmod>
  </url>
</urlset>

Parse it, compare each lastmod against the timestamp of your last successful crawl for that URL, and fetch only the survivors:

import xml.etree.ElementTree as ET
from datetime import datetime, timezone

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def stale_urls(sitemap_xml: bytes, last_success: dict):
    """Yield URLs whose lastmod is newer than our last successful crawl."""
    root = ET.fromstring(sitemap_xml)
    for entry in root.findall("sm:url", NS):
        loc = entry.find("sm:loc", NS).text.strip()
        mod = entry.find("sm:lastmod", NS)
        if mod is None:
            yield loc                    # no timestamp: assume stale, fetch it
            continue
        ts = datetime.fromisoformat(
            mod.text.replace("Z", "+00:00")
        ).astimezone(timezone.utc)
        if ts > last_success.get(loc, datetime.min.replace(tzinfo=timezone.utc)):
            yield loc

Note the safe default: a missing lastmod means fetch. Silence is not evidence of no change.

Now the caveat, and it’s a big one: lastmod honesty varies wildly. Some CMSs stamp every page on every deploy, which floods the filter with false positives — annoying but harmless. The dangerous direction is the opposite: sites that never update lastmod when content changes. Your filter silently drops real updates and you ship outdated data with total confidence. There is no protocol police enforcing this field.

So audit it. Once a month, fetch a random sample of URLs the filter called “unchanged” and hash them (next section). If real changes appear, lastmod is dead weight on that target and you fall back to fetch-then-diff. My opinion after being burned by this: treat lastmod as a hint you verify, never a contract you trust.

Content Hashing: The Ground-Truth Check for Pages That Lie About Their Timestamps

Hashing answers a different question than the first two signals. ETags and lastmod ask “does the server think it changed?” A hash asks “did the bytes change?” When validators lie, only the second question matters.

But never hash raw response bytes. Rotating ad creatives, per-request CSRF tokens, cache-busting query strings in asset URLs, and cart counters will flag half your corpus as “changed” on every cycle. Normalize first, then hash:

import hashlib
from bs4 import BeautifulSoup

DROP_TAGS = ["script", "noscript", "iframe", "svg"]
DROP_SELECTORS = [".cart-badge", "#csrf-token", "footer .cache-timestamp"]

def content_hash(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for sel in DROP_SELECTORS:
        for el in soup.select(sel):
            el.decompose()
    for tag in DROP_TAGS:
        for el in soup.find_all(tag):
            el.decompose()
    normalized = soup.decode(separator="").strip()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

page_hashes = load_hashes()  # {url: sha256_hex}

def changed(url: str, html: str) -> bool:
    h = content_hash(html)
    if page_hashes.get(url) == h:
        return False
    page_hashes[url] = h
    return True

The difference in practice, on a page where the only delta is the header cart badge incrementing from 2 to 3:

page:    https://store.example.com/products/42
trigger: cart badge 2 -> 3

raw hash:      1a77b3... != 9f2c41...  -> flagged as changed  (false positive)
normalized:    5d0e9f... == 5d0e9f...  -> unchanged           (correct)