Measure Content Density: How Much of a Page Is Data

Why Raw Page Weight Misleads: Defining Scripts, Styles, and Text as Separate Byte Buckets

A scraped page that weighs 800 KB tells you almost nothing. Was it 700 KB of JavaScript bundles and 12 KB of actual product data? Or 700 KB of dense text you can feed straight into a parser? The byte count alone can’t answer that, and every downstream decision — storage budget, parsing strategy, whether to bother with JS rendering at all — depends on the answer.

The fix is to stop treating a page as one number. Split it into buckets, measure each bucket’s byte share, and compare pages on composition rather than weight. I call this content density: the fraction of a page’s bytes that is extractable data versus everything else.

Before writing any code, pin down exactly which bytes belong to which bucket. Ambiguity here poisons every ratio you compute later.

Bucket	What counts toward it	What does NOT count
Inline scripts	Text content of <script> tags with no src	—
Referenced scripts	The <script src="..."> tag bytes themselves	The external .js file (not fetched)
Inline styles	Text content of <style> tags	—
Referenced styles	The <link rel="stylesheet"> tag bytes	The external .css file (not fetched)
HTML comments	Everything inside <!-- -->	Conditional-comment payloads are still comments
Embedded JSON-LD	Text content of <script type="application/ld+json">	Inline scripts with any other type
HTML markup	All remaining tags and attributes after the above are removed	—
Extractable text	Remaining character data, whitespace-normalized	Text inside script/style (already bucketed)

Two decisions in that table are deliberate and you might disagree with them. First, referenced scripts and styles count only their tag bytes, not the external payload. That means the analyzer undercounts what a browser actually downloads. I’m fine with that: the tool measures the HTML document you scraped and stored, not the full resource graph. If you want the browser-eye view, you need a rendering harness, which is a different tool. Second, JSON-LD gets its own bucket rather than being lumped into “text” — structured data embedded in a page is qualitatively different from prose, and mixing the two hides useful signal.

Here’s the example.com homepage, annotated by bucket:

<!doctype html>                                  <!-- markup -->
<html>                                            <!-- markup -->
<head>                                            <!-- markup -->
    <title>Example Domain</title>                 <!-- markup -->
    <meta charset="utf-8" />                      <!-- markup -->
    <style>                                       <!-- markup (the tag itself) -->
        body { margin: 0; }                       <!-- inline styles -->
    </style>                                      <!-- markup -->
</head>                                           <!-- markup -->
<body>                                            <!-- markup -->
    <h1>Example Domain</h1>                       <!-- markup + text -->
    <p>This domain is for use in illustrative     <!-- text -->
       examples in documents.</p>                 <!-- text -->
</body>                                           <!-- markup -->
</html>                                           <!-- markup -->

Eleven lines, four buckets represented. A real store page adds the other four: external script tags, external stylesheets, comments, and usually a JSON-LD block. Now let’s go collect some.

Scraping a Local Sample: Saving Raw HTML from example.com and store.example.com

The analyzer works on files, not live fetches. That keeps runs reproducible, lets you re-run the parser after tweaking bucket rules without hitting anyone’s server again, and mirrors the “store raw HTML first, parse later” pattern I argue for in Parse at Fetch Time vs Store Raw HTML. If you change your mind about what counts as text, you re-parse the same bytes instead of re-scraping.

The sampler below fetches three pages: the example.com homepage and two page types from a store.example.com storefront — the homepage and a product page. Different page types matter here, because density varies wildly between them and averaging across one page type lies to you.

# sampler.py
import time
import requests
from pathlib import Path

SAMPLE_DIR = Path("sample")
SAMPLE_DIR.mkdir(exist_ok=True)

TARGETS = {
    "example_home": "https://example.com/",
    "store_home": "https://store.example.com/",
    "store_product": "https://store.example.com/products/widget-42",
}

HEADERS = {
    "User-Agent": "density-analyzer/0.1 (contact: you@example.com)",
    "Accept": "text/html,application/xhtml+xml",
}

for name, url in TARGETS.items():
    resp = requests.get(url, headers=HEADERS, timeout=30)
    resp.raise_for_status()
    path = SAMPLE_DIR / f"{name}.html"
    path.write_bytes(resp.content)
    print(f"saved {name}: {len(resp.content)} bytes from {url}")
    time.sleep(3)  # polite delay between requests

Notes on the code: resp.content writes the raw bytes rather than resp.text, because resp.text applies a decoded (and possibly wrong) charset. You want the exact bytes you’d store. The 3-second delay is arbitrary but sufficient for a three-page sample against domains that aren’t yours.

A run produces:

saved example_home: 1256 bytes from https://example.com/
saved store_home: 483112 bytes from https://store.example.com/
saved store_product: 612940 bytes from https://store.example.com/products/widget-42

And the directory:

$ ls -l sample/
-rw-r--r--  1 you  staff   1256 example_home.html
-rw-r--r--  1 you  staff  483112 store_home.html
-rw-r--r--  1 you  staff  612940 store_product.html

One caveat: some storefronts serve a block page or a shell that hydrates client-side to plain requests. If your saved HTML is 4 KB of scaffold with no product data in it, you’re measuring the block page, not the page — a failure mode covered in detail in Your 200 OK Responses Are Actually Block Pages. For targets like that, fetch through a rendering-capable scraping API and save the rawHtml format it returns; the analyzer doesn’t care where the bytes came from.

Splitting a Page into Buckets with BeautifulSoup: The Core Parser Function

The parser’s job: take an HTML string, remove things in the right order, and measure each removed fragment. Order matters — if you extract text before removing script bodies, script code pollutes your text bucket, and every density number you report afterward is inflated garbage.

The sequence that works:

1. Pull out JSON-LD blocks first (they’re <script> tags, so a naive script sweep would swallow them).
2. Remove all remaining <script> tags, splitting inline content from src-referenced tags.
3. Remove <style> tags and <link rel="stylesheet"> tags the same way.
4. Strip comments.
5. Measure remaining markup by subtracting extracted text from what’s left.

# density.py
from bs4 import BeautifulSoup, Comment

def decompose(html: str) -> dict:
    soup = BeautifulSoup(html, "html.parser")

    def utf8_len(fragment) -> int:
        return len(fragment.encode("utf-8"))

    buckets = {
        "scripts_inline": 0,
        "scripts_ref": 0,
        "styles_inline": 0,
        "styles_ref": 0,
        "comments": 0,
        "jsonld": 0,
        "markup": 0,
        "text": 0,
    }

    # 1. JSON-LD first, before the generic script sweep
    for tag in soup.find_all("script", type="application/ld+json"):
        buckets["jsonld"] += utf8_len(tag.string or "")
        tag.decompose()

    # 2. Scripts: inline content vs the tag itself for src references
    for tag in soup.find_all("script"):
        if tag.has_attr("src"):
            buckets["scripts_ref"] += utf8_len(str(tag))
        else:
            buckets["scripts_inline"] += utf8_len(tag.string or "")
        tag.decompose()

    # 3. Styles: <style> content vs <link rel=stylesheet> tags
    for tag in soup.find_all("style"):
        buckets["styles_inline"] += utf8_len(tag.string or "")
        tag.decompose()
    for tag in soup.find_all("link", rel=lambda v: v and "stylesheet" in v):
        buckets["styles_ref"] += utf8_len(str(tag))
        tag.decompose()

    # 4. Comments
    for c in soup.find_all(string=lambda s: isinstance(s, Comment)):
        buckets["comments"] += utf8_len(c)
        c.extract()

    # 5. Remaining: markup + text
    text = soup.get_text()
    normalized = " ".join(text.split())
    buckets["text"] = utf8_len(normalized)
    buckets["markup"] = utf8_len(str(soup)) - buckets["text"]

    buckets["total"] = sum(
        buckets[k] for k in buckets if k != "total"
    )
    return buckets

A few implementation details worth defending:

• utf8_len re-encodes every fragment before len(). BeautifulSoup hands you Python strings; len() on those counts characters, not bytes. A page with heavy non-ASCII content (prices in €, CJK product names) would report character counts in the text bucket and byte counts nowhere. Encode everything to UTF-8 or your buckets are measured in mixed units.
• tag.decompose() rather than extract() for things you never need again — decompose frees the underlying tree nodes, which matters when you run this over thousands of files in one process.
• markup is computed as a residual: serialize what’s left, subtract the text bytes. This sidesteps the fiddly alternative of walking every remaining tag and summing str(tag) lengths, which double-counts nested tags.

Running it on the example.com homepage:

html = Path("sample/example_home.html").read_text(encoding="utf-8")
print(decompose(html))

{'scripts_inline': 0, 'scripts_ref': 0, 'styles_inline': 76,
 'styles_ref': 0, 'comments': 0, 'jsonld': 0,
 'markup': 941, 'text': 239, 'total': 1256}

The homepage is 1,256 bytes: 941 bytes of markup, 239 bytes of normalized text, 76 bytes of inline CSS. No scripts at all. Roughly 19% of the document is extractable text — unusually high, because there is nothing else on the page. Compare that to the store pages and the picture inverts hard.

Counting Text Bytes Honestly: Whitespace Normalization and Encoding Pitfalls

The text bucket is where most analyzers quietly cheat themselves. soup.get_text() returns the concatenation of all text nodes, including the indentation and newlines the HTML author used for readability. Those bytes are not data. Nobody downstream wants them. But if you count them, your density ratio flatters every pretty-printed page on the internet.

Here’s the wrong way and the right way, side by side:

# WRONG: counts indentation and newlines as "text"
raw_text = soup.get_text()
text_bytes_wrong = len(raw_text.encode("utf-8"))

# RIGHT: collapse all whitespace runs to single spaces, then encode
normalized = " ".join(raw_text.split())
text_bytes_right = len(normalized.encode("utf-8"))

" ".join(s.split()) does more than collapse runs: it strips leading/trailing whitespace and normalizes tabs and newlines in one call. It’s the one-liner I reach for unless a use case genuinely needs to preserve paragraph boundaries, in which case you normalize per text node and join with a separator you count separately.

On the store.example.com product page, the difference is not subtle. A before/after measurement from a local run:

store_product.html
  naive get_text():        118,404 bytes
  normalized:              14,271 bytes
  phantom whitespace:      104,133 bytes (88% of the naive count)

Without normalization, the product page would report a text density near 19%. With it, the honest number is about 2.3%. That’s an order-of-magnitude error introduced by one line of code, and it flows straight into any decision you make based on the ratio.

The encoding pitfall is sneakier. If the page declares charset=iso-8859-1 and you read the file with Path.read_text(encoding="utf-8"), you either crash or mojibake your way into wrong byte counts. Read files as bytes and let BeautifulSoup handle decoding from the document’s own meta declaration — it’s what the parser is for. If you want to go deeper on that class of bug, see Garbled Scraped Text: Fix Character Encoding Before Parsing.

Computing Density Ratios per Page and Averaging Across the Sample

Raw bucket counts answer “how many bytes.” Ratios answer “what is this page made of.” Compute both per page, then aggregate — but aggregate carefully. Mean density across a sample where one page is a 600 KB product page and two are tiny is dominated by the big page. Report median alongside mean, and per-page numbers alongside both, or your summary hides exactly the variation you sampled multiple page types to find.

# analyze.py
import csv
import statistics
from pathlib import Path
from density import decompose

BUCKETS = ["scripts_inline", "scripts_ref", "styles_inline",
           "styles_ref", "comments", "jsonld", "markup", "text"]

def analyze_sample(sample_dir: str, out_csv: str):
    rows, per_bucket_pct = [], {b: [] for b in BUCKETS}

    for path in sorted(Path(sample_dir).glob("*.html")):
        html = path.read_bytes().decode("utf-8", errors="replace")
        b = decompose(html)
        total = b["total"] or 1
        row = {"file": path.name, "total_bytes": b["total"]}
        for bucket in BUCKETS:
            pct = round(100 * b[bucket] / total, 2)
            row[f"{bucket}_pct"] = pct
            per_bucket_pct[bucket].append(pct)
        rows.append(row)

    with open(out_csv, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)

    print(f"{'bucket':<16}{'mean %':>10}{'median %':>12}")
    for bucket in BUCKETS:
        vals = per_bucket_pct[bucket]
        print(f"{bucket:<16}"
              f"{statistics.mean(vals):>10.2f}"
              f"{statistics.median(vals):>12.2f}")

analyze_sample("sample", "density_report.csv")

The CSV it writes for the three-page sample:

file,total_bytes,scripts_inline_pct,scripts_ref_pct,styles_inline_pct,styles_ref_pct,comments_pct,jsonld_pct,markup_pct,text_pct
example_home.html,1256,0.0,0.0,6.05,0.0,0.0,0.0,74.92,19.03
store_home.html,483112,41.87,1.12,7.94,0.21,0.44,0.0,37.61,10.81
store_product.html,612940,38.62,1.31,6.71,0.18,0.31,1.02,25.14,2.33

Read the store_product row: 38.6% inline scripts, 25.1% markup, 2.3% text. Three quarters of the document is markup and script. The JSON-LD block on the product page is 1% of bytes — small in share, but it’s likely the single densest extractable structure on the page, which is a good reminder that density share and data value are related but not identical.

Reading the Results: A Minimal Homepage vs. a Script-Heavy Store Page

Put the two extremes side by side:

Metric	example.com homepage	store.example.com product page
Total size	1.2 KB	598.6 KB
% scripts (inline + ref)	0.00%	39.93%
% styles (inline + ref)	6.05%	6.89%
% markup	74.92%	25.14%
% extractable text	19.03%	2.33%
% JSON-LD	0.00%	1.02%

The console summary from the analyzer’s per-page block:

=== store_product.html ===
total      612,940 bytes
scripts    39.93%  (inline 38.62%, referenced 1.31%)
styles      6.89%
markup     25.14%
comments    0.31%
jsonld      1.02%
text        2.33%  (14,271 bytes normalized)

What the numbers actually tell you:

The product page is 40% script bytes by document weight. If your pipeline stores raw HTML, you’re paying storage and egress for script bodies you will never parse. That cost compounds at scale in ways Price the Bytes: A Storage Budget for Scraped Data works through. The obvious counter-move — extract at fetch time and store only the text — trades flexibility for size, and it’s the wrong trade if your extraction rules are still changing.

Text density of 2.3% is a parsing-strategy signal, not just a curiosity. At that density, CSS-selector extraction is cheap and precise; LLM extraction over the full raw HTML means paying to process 97.7% boilerplate to reach 2.3% signal. Trimming the page before it reaches any model — or requesting only_main_content at fetch time — cuts the expensive input by an order of magnitude. This is the same argument as Trim Scraped Pages Before They Flood the Agent Context, but now you have the measurement that justifies it instead of a hunch.

The homepage’s 75% markup share is mostly head boilerplate. Meta tags, charset declarations, the doctype. A 1.2 KB page has no room for the ratio to look good. Small pages will always look markup-heavy; that’s fine, because density comparisons are meaningful between pages of the same type, not across types.

An opinion I’ll defend: density ratio is a better KPI than page weight for deciding where parsing effort goes. Two pages at 600 KB are equivalent by weight. One at 40% scripts and one at 40% text are completely different engineering problems. Weight tells you what you’ll store; composition tells you what you’ll do with it.

Packaging the Analyzer as a Config-Driven CLI for Repeated Runs

Hardcoded URL lists and bucket settings get stale the moment you point the tool at a second site. Wrap the fetch-parse-report pipeline in a CLI driven by a JSON config so a new measurement campaign is a new config file, not a code edit.

// density_config.json
{
  "targets": {
    "example_home": "https://example.com/",
    "store_home": "https://store.example.com/",
    "store_product": "https://store.example.com/products/widget-42"
  },
  "sample_dir": "sample",
  "fetch": {
    "user_agent": "density-analyzer/0.1 (contact: you@example.com)",
    "delay_seconds": 3,
    "timeout": 30
  },
  "buckets": {
    "count_jsonld_separately": true,
    "count_reference_tags_only": true,
    "normalize_text_whitespace": true
  },
  "output": {
    "csv": "density_report.csv",
    "markdown_table": true
  }
}

The CLI:

# run_density.py
import argparse
import csv
import json
import time
from pathlib import Path

import requests
from density import decompose

def fetch_sample(config):
    sample_dir = Path(config["sample_dir"])
    sample_dir.mkdir(exist_ok=True)
    fetch_cfg = config["fetch"]
    headers = {"User-Agent": fetch_cfg["user_agent"]}
    for name, url in config["targets"].items():
        out = sample_dir / f"{name}.html"
        if out.exists():
            print(f"skip {name}: already on disk")
            continue
        resp = requests.get(url, headers=headers,
                            timeout=fetch_cfg["timeout"])
        resp.raise_for_status()
        out.write_bytes(resp.content)
        print(f"saved {name}: {len(resp.content)} bytes")
        time.sleep(fetch_cfg["delay_seconds"])

def report(config):
    rows = []
    for path in sorted(Path(config["sample_dir"]).glob("*.html")):
        b = decompose(path.read_bytes().decode("utf-8", errors="replace"))
        total = b["total"] or 1
        rows.append({
            "file": path.name,
            "total": b["total"],
            "scripts_pct": round(100 * (b["scripts_inline"] + b["scripts_ref"]) / total, 2),
            "styles_pct": round(100 * (b["styles_inline"] + b["styles_ref"]) / total, 2),
            "markup_pct": round(100 * b["markup"] / total, 2),
            "text_pct": round(100 * b["text"] / total, 2),
        })

    csv_path = config["output"]["csv"]
    with open(csv_path, "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=rows[0].keys())
        w.writeheader(); w.writerows(rows)

    if config["output"].get("markdown_table"):
        cols = ["file", "total", "scripts_pct", "styles_pct", "markup_pct", "text_pct"]
        print("| " + " | ".join(cols) + " |")
        print("|" + "---|" * len(cols))
        for r in rows:
            print("| " + " | ".join(str(r[c]) for c in cols) + " |")

def main():
    parser = argparse.ArgumentParser(description="Content density analyzer")
    parser.add_argument("config", help="path to JSON config")
    parser.add_argument("--refetch", action="store_true",
                        help="ignore cached sample files and re-download")
    args = parser.parse_args()

    config = json.loads(Path(args.config).read_text())
    if args.refetch:
        for p in Path(config["sample_dir"]).glob("*.html"):
            p.unlink()
    fetch_sample(config)
    report(config)

if __name__ == "__main__":
    main()

Invocation:

python run_density.py density_config.json

Output:

| file | total | scripts_pct | styles_pct | markup_pct | text_pct |
|---|---|---|---|---|---|
| example_home | 1256 | 0.0 | 6.05 | 74.92 | 19.03 |
| store_home | 483112 | 43.0 | 8.15 | 37.61 | 10.81 |
| store_product | 612940 | 39.93 | 6.89 | 25.14 | 2.33 |

The --refetch flag and the skip-if-cached logic in fetch_sample encode the same principle: fetching is the expensive, impolite step; parsing is cheap and repeatable. Cache the bytes, iterate on the parser.

Gotchas and Next Steps

A few things that will bite you:

Parser choice changes markup counts. html.parser and lxml serialize the tree slightly differently, so the residual markup bucket shifts by a few percent between parsers. Pick one and stay consistent across runs, or your trend lines are noise.

str(soup) after decompose is not byte-identical to the input. BeautifulSoup normalizes attribute quoting and tag casing. The total bucket is the sum of measured buckets, not the original file size — deliberately. Small drift between file size on disk and measured total is expected; large drift means a bucket rule is wrong.

SPA shells will read as script-heavy with near-zero text. That’s a correct measurement of the document and a misleading measurement of the rendered page. If the data you want only exists post-hydration, fetch a rendered version and analyze that — the bucket definitions don’t change, only the input does.

Don’t average across page types. The sample here mixes a homepage and product pages precisely to show the spread. In production, group by page type and report density per group.

Next steps worth taking: run the analyzer over a real crawl sample and sort pages by text density — the bottom decile is your list of candidates for only_main_content fetching or pre-fetch trimming. Feed the per-page text-byte counts into your cost ledger so storage forecasts use signal bytes, not raw bytes. And if you’re pushing pages into an LLM pipeline, measure density before and after trimming; the ratio drop is the number that justifies the extra step.