Tutorial 12 min read

Stop Cleaning HTML by Hand: Scrape Pages as Markdown

Chunking raw HTML for LLM pipelines wastes tokens and breaks parsers. See how one scrape request field returns clean, ready-to-chunk markdown instead.

FE
FineData Engineering · Editorial Policy
|

Why Raw HTML Is a Terrible Input for LLM Chunking

The symptoms are always the same. Your RAG demo answers a pricing question with the shipping policy. Your vector store fills up with chunks that all look identical. And the embedding bill for a 200-page catalog is somehow triple what the actual content justifies.

The cause sits upstream of the model: you are feeding it HTML.

Fetch a product page with a plain HTTP client and you get roughly 40KB of markup. Here is a trimmed-down fragment of what actually arrives — nested containers, a nav menu, inline scripts, and one paragraph of real content buried in the middle:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Widget Pro | store.example.com</title>
  <style>.nav{display:flex}/* ...6KB more CSS... */</style>
  <script>window.__INITIAL_STATE__={"cart":null,"exp":"hero-v3"}/* ...9KB more JS... */</script>
</head>
<body>
  <div class="page">
    <div class="header">
      <nav class="main-nav">
        <ul>
          <li><a href="/electronics">Electronics</a></li>
          <li><a href="/widgets">Widgets</a></li>
          <li><a href="/cart">Cart (0)</a></li>
        </ul>
      </nav>
    </div>
    <main>
      <div class="product">
        <div class="product__info">
          <h1>Widget Pro</h1>
          <p class="price">$49.00 <s>$59.00</s></p>
          <p>The only widget you'll ever need: machined aluminum body,
             2-year warranty, ships in recyclable packaging.</p>
        </div>
      </div>
    </main>
    <footer>© store.example.com — About — Privacy — Terms</footer>
  </div>
</body>
</html>

One paragraph of signal. Everything else is chrome.

The token math on the full 40KB page, using the usual chars-divided-by-four estimate:

MetricRaw HTMLMarkdown
Transfer size~40 KB~3 KB
Estimated tokens~10,200~780
Tokens that are actual product content~600~720
Boilerplate share~94%under 5%

Two downstream failures follow. Cost: you pay to embed roughly 9,500 junk tokens per page. Quality: chunk that document into fixed windows and most windows are nav, footer, or half a script blob. Worse, the same nav appears on every page of the site, so your index accumulates thousands of near-duplicate boilerplate embeddings — and nearest-neighbor search loves to return them. You tune the embedder. The bug is in ingestion.

The Hidden Failure Modes of Hand-Rolled HTML Cleaning

The obvious fix is to strip the tags yourself. Two classic implementations, both broken in instructive ways.

First, the greedy regex:

import re

html = open("widget.html").read()

# Greedy: matches from the FIRST '<' to the LAST '>' in the document.
text = re.sub(r"<.*>", " ", html, flags=re.S)
print(repr(re.sub(r"\s+", " ", text).strip()))

Output: ' '. Everything is gone — both prices included — because the entire document lives between the first < and the last >.

The non-greedy version survives, sort of:

text = re.sub(r"<(script|style)\b.*?</\1>", "", html, flags=re.S | re.I)
text = re.sub(r"<[^>]+>", "", text)
print(re.sub(r"\s+", " ", text)[:320])

Skip to content Menu Electronics Widgets All products Cart (0) Search Widget
Pro Add to cartRelated productsReviews (12) In stock Description The only
widget you'll ever need... Specifications Weight340 gWarranty2 years Related
products Widget Mini$29.00Widget XL$79.00 About Privacy Terms

Adjacent UI strings fuse into gibberish — Add to cartRelated productsReviews — because tags separated the elements and you deleted the tags. Table cells merge: Weight340 gWarranty2 years. Product, cross-sells, and footer collapse into one undifferentiated stream.

BeautifulSoup’s get_text() produces the same fusion, just more politely. Its default separator is an empty string, it has no concept of “main content”, and it cannot tell a <nav> from an article. There is a harder failure, too: when prices are injected by client-side JavaScript, the fetched HTML never contained them. No cleaner can recover text that was never in the response — you need a rendered page, which is a different and more expensive problem.

The deepest issue is maintenance. Every site’s markup is its own dialect, so per-site cleaning rules multiply until your “parser” is a config file of exceptions. That is the worst kind of technical debt in scraping: it fails silently, as slightly-worse chunks instead of crashes, so nobody gets paged and nobody fixes it.

One Request Field That Returns Markdown Instead of HTML

Scraping APIs have quietly converged on a better answer: convert the page server-side and return the markdown. The examples below hit FineData’s scrape endpoint, but every major scraping service has grown an option like this — what matters is the field name and what it guarantees.

On POST /api/v1/scrape, the field is formats:

curl -X POST https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer fd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/products/widget",
    "formats": ["markdown"],
    "only_main_content": true
  }'

Same call in Python:

import requests

resp = requests.post(
    "https://api.finedata.ai/api/v1/scrape",
    headers={"Authorization": "Bearer fd_your_api_key"},
    json={
        "url": "https://store.example.com/products/widget",
        "formats": ["markdown"],
        "only_main_content": True,
    },
    timeout=180,
)
resp.raise_for_status()

Side by side, abridged responses. With rawHtml requested:

{
  "success": true,
  "url": "https://store.example.com/products/widget",
  "rawHtml": "<!DOCTYPE html><html lang=\"en\"><head><meta charset=\"utf-8\"><title>Widget Pro | store.example.com</title><style>.nav{display:flex}...</style><script>window.__INITIAL_STATE__=..."
}

With markdown requested:

{
  "success": true,
  "url": "https://store.example.com/products/widget",
  "markdown": "# Widget Pro\n\n$49.00 ~~$59.00~~\n\nThe only widget you'll ever need: machined aluminum body, 2-year warranty, ships in recyclable packaging.\n\n## Specifications\n\n| Weight | 340 g |\n| --- | --- |\n| Warranty | 2 years |"
}

The conversion happens before the payload reaches your network. Headings, lists, links, tables, and code fences survive; scripts, styles, hidden divs, and layout containers do not. Adding only_main_content: true strips nav, footer, and sidebar, so the markdown starts at the product, not the menu.

One honest trade-off: you give up control of the conversion. Your downstream code now depends on the API’s markdown dialect rather than your own normalization rules. In practice that dialect is standard markdown, which your chunker and embedder handle anyway — but if you need exotic preprocessing, you are applying it to someone else’s output.

From Scrape Response to Clean Chunks in Under 20 Lines

Here is the full pipeline: fetch markdown for a category page, split on headings, print chunk previews.

import re
import requests

API = "https://api.finedata.ai/api/v1/scrape"
AUTH = {"Authorization": "Bearer fd_your_api_key"}

def scrape_markdown(url):
    r = requests.post(API, headers=AUTH, timeout=180,
                      json={"url": url, "formats": ["markdown"],
                            "only_main_content": True})
    r.raise_for_status()
    return r.json()["markdown"]

def chunk_by_heading(md, max_chars=2000):
    sections = re.split(r"\n(?=#{1,6} )", md)  # split before any heading
    chunks, buf = [], ""
    for s in sections:
        if buf and len(buf) + len(s) > max_chars:
            chunks.append(buf.strip())
            buf = s
        else:
            buf = f"{buf}\n{s}" if buf else s
    if buf.strip():
        chunks.append(buf.strip())
    return chunks

md = scrape_markdown("https://store.example.com/products?page=1")
for i, c in enumerate(chunk_by_heading(md)):
    print(f"chunk {i:02d} ({len(c):5d} chars) :: {c[:140].replace(chr(10), ' | ')}")

Twenty lines, counting blanks. No HTML parser, no selector config, no cleaning regexes to rot.

Sample output:

chunk 00 ( 1812 chars) :: # All Products | ## Widget Pro | $49.00 ~~$59.00~~ | The only widget you'll ever need: machined aluminum body, 2-year warranty...
chunk 01 ( 1644 chars) :: ## Widget Mini | $29.00 | Compact version with polymer body and 1-year warranty. Same mounting standard as Widget Pro...
chunk 02 ( 1498 chars) :: ## Travel Router Mini | $39.00 | Pocket-sized travel router with dual-band Wi-Fi, USB-C power, and a 6-hour battery...
chunk 03 (  812 chars) :: ## Cable Organizer 3-Pack | $12.00 | Silicone cable clips with adhesive backing. Keep desk cables routed and untangled...

Chunk boundaries are semantic — they follow headings, not byte offsets. Every chunk opens with a product name and carries its price and description intact. That is exactly what the embedder should see.

A style call I will defend: heading-based splits beat fixed-size windows for catalog and docs pages. Fixed windows slice mid-table and orphan prices from their product names. Heading splits keep each unit whole. For pages with no headings at all, fall back to paragraph splits and accept the trade-off.

Measuring the Difference: Tokens, Chunk Quality, and Retrieval Accuracy

Do not take my word for any of this — measure your own corpus. The harness: scrape ten pages, build two chunk sets (strip-the-HTML pipeline vs. markdown pipeline), retrieve with the same method for both, and check whether the top-3 contains a chunk holding a known fact.

A cheap scorer using TF-IDF instead of a hosted embedder — no API keys, and plenty accurate for comparing pipelines against each other:

from sklearn.feature_extraction.text import TfidfVectorizer

FACTS = {
    "Widget Pro": "$49.00",
    "Widget Mini": "$29.00",
    "Travel Router Mini": "$39.00",
    "Cable Organizer 3-Pack": "$12.00",
    # ...six more ground-truth pairs from the ten scraped pages
}

def top_k(chunks, query, k=3):
    vecs = TfidfVectorizer(stop_words="english").fit_transform(chunks + [query])
    sims = (vecs[:-1] @ vecs[-1].T).toarray().ravel()
    ranked = sims.argsort()[::-1][:k]
    return [chunks[i] for i in ranked]

def hit_rate(chunks):
    hits = sum(any(fact in c for c in top_k(chunks, f"{name} price"))
               for name, fact in FACTS.items())
    return hits, len(FACTS)

One sample run against ten store.example.com product pages. Small sample — trust the direction, not the decimals:

MetricHTML pipeline (strip + chunk)Markdown pipeline
Tokens embedded~98,000~7,400
Chunks produced11214
Avg tokens per chunk875530
Chunks that are >80% boilerplate710
Facts retrieved in top-3 (10 checks)6/1010/10

The hit-rate gap has a mundane cause. In the HTML pipeline, queries like “Widget Pro price” match boilerplate chunks — full of words like “price”, “product”, “buy” — just as strongly as the real answer, so the top-3 fills with nav and cross-sell noise before it reaches the description. In the markdown pipeline, those chunks simply do not exist.

Which leads to a claim you are free to disagree with: most “our embeddings are bad” debugging sessions are ingestion bugs. The model never saw a usable sentence in the first place. Measure ingestion before you touch models; it is cheaper and it is usually where the problem lives.

As a side effect, the markdown pipeline cut embedded tokens by roughly 13x. On a catalog large enough for the embedding bill to matter, that alone pays for the whole approach.

Handling Edge Cases: Tables, Code Blocks, and Paginated Listings

Markdown output earns its keep precisely where naive cleaning destroys the most value.

Tables. HTML-collapsed text flattens the grid into word soup:

Basic $19/mo 3 Pro $49/mo 10 Enterprise Contact us Unlimited

The markdown output keeps the structure:

| Plan       | Price      | Seats     |
|------------|------------|-----------|
| Basic      | $19/mo     | 3         |
| Pro        | $49/mo     | 10        |
| Enterprise | Contact us | Unlimited |

For a retrieval system, that pipe table is the difference between “which plan allows 10 seats?” being answerable and being a coin flip. The same applies to spec sheets and comparison matrices — anywhere column semantics carry meaning.

Code blocks. Documentation pages keep their fenced blocks:

```bash
curl -H "Authorization: Bearer ..." https://api.example.com/v1/ping
```

Without them, shell samples become indistinguishable from prose and pollute every chunk they land in.

Paginated listings need orchestration rather than parsing. For five pages, a synchronous loop over POST /api/v1/scrape is fine. Past a few dozen pages, submit a batch and let the webhook tell you when everything landed. Here is the batch submission:

import requests

AUTH = {"Authorization": "Bearer fd_your_api_key"}
BASE = "https://api.finedata.ai"

pages = [{"url": f"https://store.example.com/products?page={p}",
          "formats": ["markdown"], "only_main_content": True}
         for p in range(1, 6)]

r = requests.post(
    f"{BASE}/api/v1/async/batch",
    headers=AUTH,
    timeout=60,
    json={
        "callback_url": "https://api.example.com/hooks/scrape-done",
        "requests": pages,
    },
)
batch_id = r.json()["batch_id"]

# Poll instead if you don't run a webhook receiver:
r = requests.get(
    f"{BASE}/api/v1/async/batch/{batch_id}",
    headers=AUTH,
    params={"include_results": True},
    timeout=60,
)
print(r.json()["status"])

The webhook fires once, when every job in the batch has finished — one submission, one notification, no polling loop in your worker. For retry policies and failure handling at thousands of pages, see Async Scraping at Scale: Jobs, Batches, Webhooks.

Three gotchas while you are here. If prices appear only after JavaScript runs, add "use_js_render": true — otherwise markdown is generated from the pre-render DOM and the price column comes back empty. If you set raw_output: true, pass exactly one format; the clean-output mode rejects multi-format requests. And if a heavy page times out at the default, raise timeout before you blame the target site.

When You Still Need Raw HTML (and How to Get Both)

Markdown is not a universal replacement, and pretending otherwise will bite you in four places:

  • JSON-LD and microdata. Product schema lives in <script type="application/ld+json"> tags. Markdown drops scripts by design, so if you mine structured data, request rawHtml.
  • Link attributes. Markdown preserves links as []() but discards rel, target, and everything else you might filter on when building a link graph.
  • Debugging. When the markdown looks wrong, you want the source HTML next to it to see why.
  • Visual layout. Pixel positions and CSS state simply cannot be represented.

There is a better option than shipping HTML home and parsing it yourself: push the extraction server-side. extract_rules takes CSS/XPath selectors, and extract_schema takes a JSON Schema for AI-driven extraction — both return structured fields instead of a DOM for you to walk. The trade-offs are covered in From HTML to JSON: Schema-Driven Extraction.

When you genuinely need both representations, one request returns both:

r = requests.post(
    "https://api.finedata.ai/api/v1/scrape",
    headers={"Authorization": "Bearer fd_your_api_key"},
    json={
        "url": "https://example.com/pricing",
        "formats": ["markdown", "rawHtml"],
    },
    timeout=180,
)
body = r.json()
md, html = body["markdown"], body["rawHtml"]

Decision checklist:

Pipeline needRequest
RAG chunks / embeddingsmarkdown
Full-text search indexmarkdown or text
LLM summarizationmarkdown
JSON-LD / microdata miningrawHtml
CSS/XPath selector logicrawHtml, or extract_rules server-side
Layout debugging, visual checksrawHtml
Structured fields without DOM workextract_schema / extract_rules

My default: markdown for most pages, rawHtml only for the few that carry parseable structured data — and fetched in the same request, not as a second round trip.

Wrap-Up

Raw HTML is roughly 94% boilerplate. Hand-rolled cleaning fuses UI strings, mangles tables, and quietly rots into per-site exceptions. A single formats: ["markdown"] field moves the conversion server-side, so ingestion becomes a heading split and a print statement.

Next steps: run the twenty-line pipeline against your ten most valuable pages, score it with the fact-checking harness above, and only reach for rawHtml when you can name the exact DOM node you need. Measure your own numbers — then go delete the regex cleaner before anyone has to maintain it.

#markdown #llm-pipelines #rag #web-scraping-api #data-cleaning #slot:api-capability

Related Articles

Tutorial

Scrape Localized Storefronts With a Country Exit Code

Storefronts change price, language, and stock by visitor country. Pin the scrape exit with an ISO-2 code and handle 422 when the country is unsupported.

 Tutorial

Route Scraping Traffic Through Proxies You Already Own

When targets allowlist your IPs or you already pay for residential proxies, attach a proxy profile so scrape requests exit through your pool.

 Tutorial

Keep One Exit IP Across Multi-Step Scrape Requests

Learn how proxy_sticky keeps the same exit IP for warmup and follow-up requests, preventing session resets during multi-step scraping workflows.