Trim Scraped Pages Before They Flood the Agent Context

Why Raw Scraped HTML Overwhelms an Agent Context Window

Here is a measurement I run whenever someone tells me their MCP-connected agent “just needs the page.” On my machine:

$ curl -s https://example.com | wc -c
1256

That is the tiniest page on the internet, and it is already over a kilobyte. Real pages are not example.com. A typical product page or documentation article ships 150–400 KB of HTML, and most of it is stuff no agent ever needs: inline JSON state, tracking scripts, SVG icons, repeated navigation markup, and enough CSS class names to fill a phone book.

Page	Raw HTML (bytes)	Visible text (bytes)	Ratio
example.com	1,256	511	2.5x
example.com/docs/guide (long article)	214,000	31,000	6.9x
store.example.com/product/123	387,000	9,400	41x
store.example.com/category (grid page)	512,000	22,000	23x

That 41x ratio on the product page is the whole problem. If you pipe the raw HTML of that page into a tool response, you are paying for 387 KB of bytes to deliver 9 KB of information. And the agent does not just pay once — tool output sits in the conversation history, so it gets re-processed on every subsequent turn until the history is truncated.

A rough estimate of the damage, using the standard heuristic of ~4 characters per token for prose and ~3 for markup-heavy text:

def estimate_tokens(text: str, chars_per_token: float = 3.5) -> int:
    """Rough token estimate. Use tiktoken for exact counts if you bill on tokens."""
    return int(len(text) / chars_per_token)

raw_html = open("product_page.html").read()
print(estimate_tokens(raw_html))          # ~110,000 tokens for 387 KB
print(estimate_tokens(visible_text))      # ~2,700 tokens for 9.4 KB

One product page, untrimmed, eats most of a 128k context window. The same page trimmed costs under 3k. That is the gap this pipeline closes.

Sizing Your Context Budget Before Choosing a Trimming Strategy

Do not pick a “max output size” by vibes. Work backwards from the model’s window and subtract everything that is not page content.

For a 128k-token model serving an agent with a handful of MCP tools, a budget I have found workable looks like this:

Budget slot	Tokens	Notes
System prompt	2,000	Agent persona, instructions, safety rules
MCP tool definitions	6,000	Schemas for ~10 tools; grows fast with verbose descriptions
Multi-turn history	40,000	User messages, reasoning, prior tool calls
Tool output (all tools combined)	24,000	The slot we are engineering for
Response headroom	20,000	Room for the model to actually answer
Safety margin	36,000	Long tool outputs, schema drift, unexpected history growth

The important number is that 24k tool-output slot. It is shared. If your scrape tool returns 20k tokens for one page, every other tool call in that conversation is fighting over 4k. So the real per-page budget is not 24k — it is more like 6–8k tokens, assuming the agent might scrape two or three pages in one task.

Here are the constants the rest of this tutorial uses:

# Budget configuration -- tune to your model and tool count
MAX_TOOL_OUTPUT_TOKENS = 6_000   # hard cap per tool response
CHARS_PER_TOKEN = 3.5            # heuristic for markdown-ish text
MAX_TOOL_OUTPUT_CHARS = MAX_TOOL_OUTPUT_TOKENS * CHARS_PER_TOKEN  # 21,000
CHUNK_SIZE = 18_000              # chars per chunk, leaves room for metadata
OVERLAP = 1_500                  # chars of overlap between chunks
MAX_PAGE_CHUNKS = 4              # refuse to paginate beyond this

Why MAX_PAGE_CHUNKS = 4? Because if a trimmed page still needs more than four chunks, the page is not the problem — your extraction is. A product page or article that produces 70k+ characters of trimmed content usually means the readability pass failed and you are passing through boilerplate. Fail loudly instead of silently flooding context. This is an opinion, and you might disagree: if your agent genuinely reads 50-page documentation guides, raise the cap. But make it a deliberate number, not an accident.

Stripping Boilerplate: Selecting Main Content with a Readability Pass

The first transformation is removing everything that is not the article body: nav bars, script tags, footers, cookie banners, related-item carousels.

I do this at fetch time rather than after storing the HTML, for reasons covered in Parse at Fetch Time vs Store Raw HTML — but the short version is that raw HTML you never needed is dead weight in storage too. The scraping API I use exposes only_main_content, which strips navigation, footer, and sidebar markup server-side before the response ever reaches my code. That handles most of the work. For pages where I want finer control, I run a readability-style pass locally on top.

import requests
from readability import Document
from bs4 import BeautifulSoup

def scrape_html(url: str) -> str:
    resp = requests.post(
        "https://api.finedata.ai/api/v1/scrape",
        headers={"Authorization": "Bearer fd_your_api_key"},
        json={
            "url": url,
            "formats": ["rawHtml"],
            "only_main_content": True,   # server-side boilerplate strip
            "use_js_render": True,       # product pages are usually SPAs
        },
        timeout=180,
    )
    resp.raise_for_status()
    return resp.json()["data"]["rawHtml"]

def readability_pass(html: str) -> str:
    doc = Document(html)
    summary_html = doc.summary(html_partial=True)
    soup = BeautifulSoup(summary_html, "html.parser")

    for tag in soup(["script", "style", "noscript", "svg", "iframe", "form"]):
        tag.decompose()

    return str(soup)

raw = scrape_html("https://store.example.com/product/123")
trimmed = readability_pass(raw)
print(f"raw:     {len(raw):,} bytes")
print(f"trimmed: {len(trimmed):,} bytes")

Typical output on that product page:

raw:     387,412 bytes
trimmed: 11,203 bytes

A 97% reduction before we have done anything clever with formatting. Which approach should you use for the strip?

Approach	Output size	Fidelity	When to use
Regex strip (re.sub on <script>...</script>)	~60% of raw	Low — breaks on nested tags, malformed HTML, inline attributes	Never for production. Fine for a quick debug
DOM walk with BeautifulSoup, tag allowlist	~8–15% of raw	Medium — you must maintain the allowlist per site family	Sites with stable, known structure
Readability library (or server-side only_main_content)	~3–6% of raw	High on articles, medium on product/grid pages	Default choice; falls back to DOM walk when it mispicks the container

Readability’s weak spot is non-article layouts. On a category grid page it sometimes latches onto the filter sidebar instead of the product grid. When that happens, the DOM walk with an explicit allowlist wins. My pipeline tries readability first and falls back to a DOM walk when the readability output is suspiciously small (under 500 bytes) or suspiciously large (over half the input).

Converting Trimmed HTML to Markdown for Token-Efficient Tool Output

HTML is a bad delivery format for agents even after trimming. Tags, attributes, and class names burn tokens carrying zero semantic content, and models parse Markdown more reliably than they parse arbitrary DOM structures. There is a reason every major agent framework converged on Markdown for tool output.

import markdownify

def dom_to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")

    # Collapse whitespace-only text nodes so we don't emit blank-line runs
    for el in soup.find_all(string=True):
        if not el.strip():
            el.extract()

    md = markdownify.markdownify(str(soup), heading_style="ATX", bullets="-")
    # Squash 3+ newlines down to 2
    import re
    return re.sub(r"\n{3,}", "\n\n", md).strip()

md = dom_to_markdown(trimmed)
print(f"markdown: {len(md):,} chars, ~{estimate_tokens(md):,} tokens")

Same product page content, measured three ways:

Representation	Chars	Est. tokens	Notes
Raw HTML	387,412	~110,000	Blows the entire budget
Trimmed HTML	11,203	~3,200	Still pays for tags and attributes
Markdown	6,880	~1,970	Fits comfortably; tables and headings survive

The Markdown conversion preserves what agents actually use: heading hierarchy (##, ###) gives them a map of the page, tables stay tables, lists stay lists. The product title, price, description, and spec table on that page all survive intact at roughly 1.8% of the original token cost.

One honest caveat: Markdown conversion is lossy. If your downstream task needs exact DOM structure — say, verifying that a checkout button exists with a specific data-testid — convert to a compact JSON of selected elements instead. For “read this page and answer questions,” Markdown wins. For “assert on this page’s structure,” it does not. I have seen teams ship Markdown conversion everywhere and then wonder why their DOM-assertion tests went flaky. Different jobs, different formats.

If you would rather not run the conversion yourself, the same API accepts formats: ["markdown"] directly and returns server-converted Markdown — worth doing at scale so your worker boxes are not burning CPU on markdownify for every page. The local pipeline below keeps conversion in-process so you can see every stage.

Chunking Long Pages with Overlap So Nothing Critical Is Lost

Some pages survive trimming and conversion and are still too big. A 30k-token documentation guide is legitimately 30k tokens of content. For those, chunk — with overlap, and with metadata labels the agent can act on.

def chunk_text(text: str, chunk_size: int = CHUNK_SIZE,
               overlap: int = OVERLAP) -> list[str]:
    if len(text) <= chunk_size:
        return [text]

    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        # Prefer breaking on a paragraph boundary near the cut point
        if end < len(text):
            window = text[end - 400:end]
            cut = window.rfind("\n\n")
            if cut != -1:
                end = end - 400 + cut
        chunks.append(text[start:end])
        if end >= len(text):
            break
        start = end - overlap
    return chunks

def label_chunk(body: str, index: int, total: int,
                start: int, end: int, url: str) -> str:
    header = (
        f"[CHUNK {index + 1}/{total} | chars {start}-{end} | "
        f"source: {url}]\n\n"
    )
    return header + body

guide_md = dom_to_markdown(readability_pass(
    scrape_html("https://example.com/docs/guide")))
parts = chunk_text(guide_md)
labeled = [label_chunk(p, i, len(parts),
                       i * (CHUNK_SIZE - OVERLAP), 0,  # recompute below
                       "https://example.com/docs/guide")
           for i, p in enumerate(parts)]

The paragraph-boundary preference matters more than people expect. Cutting mid-table or mid-code-block produces chunks the agent misreads — it sees half a code sample and assumes it is complete. Snapping cuts to \n\n boundaries costs you a few hundred characters of chunk utilization and saves you from wrong answers.

What a labeled chunk looks like when it lands in the tool response:

[CHUNK 1/3 | chars 0-17940 | source: https://example.com/docs/guide]

# API Guide

## Authentication

All requests require a bearer token...

## Rate Limits

| Plan | Requests/min |
|---|---|
| Free | 60 |
| Pro | 600 |
...

The header does double duty. It tells the agent where this content sits in the whole document, and it gives the agent the vocabulary to ask for the next piece instead of hallucinating the rest. Without the label, agents routinely treat a chunk as the entire page and answer questions about content they never saw. That failure mode is silent, which makes it worse than a crash.

On overlap: 1,500 characters is my default, and I think the common advice to use 10–20% overlap is overshooting for agent tool output. Overlap exists to prevent a semantic split at a chunk boundary. For retrieval-into-RAG you want generous overlap because an embedding of half a thought is useless. For an agent reading chunks sequentially, a sentence of overlap is usually enough to re-anchor. You are paying tokens for every duplicated character on every chunk, so overlap is a budget line item, not a free safety net.

Wiring the Trim Pipeline into an MCP Tool Response

Now assemble it. The MCP tool handler below scrapes, trims, converts, chunks, and returns chunk 1 plus a continuation pointer the agent can call for more.

# mcp_server.py -- tool handler (FastMCP-style)
import json, hashlib

CHUNK_STORE: dict[str, dict] = {}  # in prod: Redis with TTL

@tool
def read_web_page(url: str, chunk: int = 1) -> dict:
    """Fetch a web page and return its main content as markdown.
    Long pages are split into labeled chunks; call again with a
    higher chunk number to continue reading."""

    raw = scrape_html(url)
    md = dom_to_markdown(readability_pass(raw))

    if estimate_tokens(md) > MAX_TOOL_OUTPUT_TOKENS * MAX_PAGE_CHUNKS:
        return {
            "error": "page_exceeds_budget",
            "detail": (
                f"Trimmed content is ~{estimate_tokens(md)} tokens, "
                f"above the {MAX_PAGE_CHUNKS}-chunk cap. Use a more "
                f"specific URL or an extract_schema request."
            ),
        }

    parts = chunk_text(md)
    if len(parts) > MAX_PAGE_CHUNKS:
        return {"error": "page_exceeds_budget",
                "detail": f"{len(parts)} chunks needed, cap is {MAX_PAGE_CHUNKS}"}

    page_id = hashlib.sha1(url.encode()).hexdigest()[:12]
    CHUNK_STORE[page_id] = {"url": url, "chunks": parts}

    body = label_chunk(parts[chunk - 1], chunk - 1, len(parts),
                       (chunk - 1) * (CHUNK_SIZE - OVERLAP),
                       chunk * CHUNK_SIZE, url)

    return {
        "content": body[:MAX_TOOL_OUTPUT_CHARS],
        "page_id": page_id,
        "chunk": chunk,
        "total_chunks": len(parts),
        "has_more": chunk < len(parts),
        "next_chunk_tool": "read_web_page",
        "next_chunk_args": {"url": url, "chunk": chunk + 1},
    }

The JSON the agent actually receives for the docs guide:

{
  "content": "[CHUNK 1/3 | chars 0-17940 | source: https://example.com/docs/guide]\n\n# API Guide\n\n## Authentication\n...",
  "page_id": "a91f3c2e77b4",
  "chunk": 1,
  "total_chunks": 3,
  "has_more": true,
  "next_chunk_tool": "read_web_page",
  "next_chunk_args": {"url": "https://example.com/docs/guide", "chunk": 2}
}

Two design decisions worth defending. First, has_more and next_chunk_args are explicit fields, not prose hints buried in the content. Agents follow structured continuation pointers far more reliably than they follow “there is more content” written in natural language. Second, the chunk store is keyed by a hash of the URL, so repeated calls for chunk 2 and 3 do not re-scrape — but that also means the content can go stale if the agent pages through over many minutes. Set a TTL on the store that matches how long a single task should run; an hour is plenty.

If you are building MCP tools seriously, read up on auth scopes and failure modes — a scraping tool that holds an API key is exactly the kind of thing that needs scoped, revocable credentials, and the key-management endpoints (POST /api/v1/api-keys, DELETE /api/v1/api-keys/{key_id}) let you issue a dedicated low-balance key per agent deployment instead of sharing your main one.

Validating the Pipeline: Measuring Real Token Savings End to End

A trimming pipeline you do not measure is a trimming pipeline that will silently regress. The first time a target site ships a redesign and readability starts grabbing the wrong container, your output sizes will tell you before your users do.

# benchmark.py
import statistics

URLS = [
    "https://example.com",
    "https://example.com/docs/guide",
    "https://example.com/docs/quickstart",
    "https://example.com/docs/api-reference",
    "https://example.com/docs/errors",
    "https://store.example.com/product/123",
    "https://store.example.com/product/456",
    "https://store.example.com/product/789",
    "https://store.example.com/category/tools",
    "https://store.example.com/about",
]

def run_pipeline(url: str) -> dict:
    raw = scrape_html(url)
    md = dom_to_markdown(readability_pass(raw))
    chunks = chunk_text(md)
    return {
        "url": url,
        "raw_tokens": estimate_tokens(raw, 3.0),
        "final_tokens": estimate_tokens(md),
        "chunks": len(chunks),
        "within_budget": estimate_tokens(md) <= MAX_TOOL_OUTPUT_TOKENS,
    }

results = [run_pipeline(u) for u in URLS]
for r in results:
    pct = 100 * (1 - r["final_tokens"] / r["raw_tokens"])
    flag = "" if r["within_budget"] else "  <-- OVER BUDGET"
    print(f"{r['url']:48} {r['raw_tokens']:>8,} -> "
          f"{r['final_tokens']:>6,} tok  ({pct:5.1f}% saved, "
          f"{r['chunks']} chunk(s)){flag}")

Sample run:

URL	Raw tokens	Final tokens	Saved	Chunks	In budget
example.com	430	190	55.8%	1	yes
example.com/docs/guide	71,300	9,850	86.2%	3	per-chunk
example.com/docs/quickstart	18,600	3,100	83.3%	1	yes
example.com/docs/api-reference	96,000	26,400	72.5%	6	no — capped
example.com/docs/errors	12,200	2,400	80.3%	1	yes
store.example.com/product/123	110,000	1,970	98.2%	1	yes
store.example.com/product/456	104,000	1,880	98.2%	1	yes
store.example.com/product/789	118,000	2,050	98.3%	1	yes
store.example.com/category/tools	146,000	6,300	95.7%	1	yes
store.example.com/about	41,000	1,100	97.3%	1	yes

The api-reference row is the interesting one. It is a 26k-token reference page and the pipeline correctly refuses to chunk it into six pieces. The fallback behavior — returning page_exceeds_budget with a suggestion to use a more specific URL — is the right call here, because paging an agent through six chunks of a flat reference table is worse than pointing it at the specific endpoint section it needs. In a richer build, that fallback would switch to extract_schema or extract_rules on the scrape request and pull only the relevant table, which is the schema-driven approach worth reading about separately.

Median savings across the run: 86%. Worst case: 55% on a page that was already tiny. That is the shape of result you should demand from any trimming pipeline — the biggest wins on the biggest pages, and no page ever exceeding the budget slot you carved out for it.

Wrap-Up

The pipeline is four stages, each with one job: strip boilerplate, convert to Markdown, chunk with labels, cap with a refusal. Run the benchmark before you trust it, and keep it in CI so a target-site redesign shows up as a failing number instead of a confused agent.

Gotchas to watch:

• Readability mispicks containers on non-article pages. Detect it with size sanity checks and fall back to a DOM allowlist.
• Chunk cuts through tables and code blocks unless you snap to paragraph boundaries. Mid-table cuts cause silent misreads.
• Stale chunk stores. TTL them, or re-scrape on continuation if freshness matters more than cost.
• Markdown is lossy. If the task needs DOM structure, emit selected elements as JSON instead.
• Overlap is a token cost, not a free option. For sequential agent reading, a sentence beats the 15% overlap default from RAG folklore.

Next steps if you are taking this further: swap the generic trim for extract_schema requests when the agent’s questions are predictable, so the server returns exactly the fields you need and skips the Markdown stage entirely. And if your agent reads pages repeatedly across sessions, cache the trimmed Markdown keyed by URL plus a content hash — re-paying scrape tokens for an unchanged page is the next leak to plug.