Serve Scraped Data to Agents as MCP Resources

Why Resources Beat Tools for Serving Scraped Snapshots

Most people wire scraped data into MCP servers as tools. The agent calls scrape_products, waits two seconds, gets a wall of JSON, and moves on. Ten minutes later it calls the same tool again because it needs the data a second time — and the server happily re-fetches the page, burns another token of scrape budget, and returns roughly the same bytes. Multiply that by a long-running agent session and your scrape bill is dominated by redundant fetches nobody asked for.

MCP has a second primitive that fits this job better: resources. A resource is a URI-addressed blob of data the server owns. Agents read resources when they need them, list them to discover what exists, and — the part almost nobody uses — subscribe to them so the server pushes a notification when the contents change. Read-oriented scraped data is exactly this shape. The scraper owns the fetching schedule. The agent just reads the latest snapshot.

Here’s the honest comparison:

	MCP tool call (re-scrape)	MCP resource read	MCP resource subscription
Trigger	Agent-initiated, every time	Agent-initiated, cached	Server-initiated on change
Cost per access	Full scrape cost each call	~0 (local lookup)	~0 (push notification)
Freshness	Fresh but redundant	As fresh as last scrape	Bounded by scrape cadence
Re-fetch behavior	Fetches on every invocation	Never fetches on read	Scraper fetches on its own schedule

The sequence looks like this:

Tool-based (agent re-invokes 5 times):
  Agent ──scrape_products()──▶ Scraper ──HTTP──▶ store.example.com
  Agent ◀──JSON (2.1s)──────── Scraper ◀──────── store.example.com
  ... repeated 5 times, 5 HTTP fetches, ~10s wall time, 5x scrape cost

Resource-based (scraper refreshes on schedule):
  Scraper ──HTTP──▶ store.example.com        (1 fetch, on cron)
  Scraper ──save snapshot + emit updated──▶ MCP server
  MCP server ──resources/updated──▶ Agent    (push)
  Agent ──resources/read──▶ MCP server ──▶   (30ms, no network)
  Agent ──resources/read──▶ MCP server ──▶   (30ms, no network)
  ... 4 more reads, 0 HTTP fetches to the target site

One fetch serves unlimited reads. If your agents touch the same pages repeatedly, this is the difference between paying per access and paying per change. I wrote about the caching variant of this problem before — stop re-fetching the same page through tool calls — but resources are the cleaner primitive: the cache is the interface, not a patch behind it.

Modeling Scrape Results as Resource URIs

A resource is only useful if its URI is stable. The agent should be able to read the same address tomorrow and get the newest data without knowing anything about your scrape schedule. I use a scheme that mirrors the source site’s structure:

scrape://store.example.com/products/latest        → newest full product listing snapshot
scrape://store.example.com/products/{id}          → newest snapshot for one product
scrape://store.example.com/products/latest?format=markdown  → same snapshot, rendered as markdown

The host part is the scraped origin, the path is the page, and latest is a reserved segment meaning “most recent version.” Versioned reads (?version=<hash>) are possible but agents rarely need them — the point is that latest is always current.

When the agent lists resources, the server returns metadata that tells it what each URI contains:

{
  "resources": [
    {
      "uri": "scrape://store.example.com/products/latest",
      "name": "store.example.com — product listing (latest)",
      "mimeType": "application/json",
      "description": "Full product listing scraped from https://store.example.com/products. Contains sku, title, price, and availability for 142 products. Snapshot taken 2026-01-30T09:15:02Z; refreshed every 30 minutes."
    },
    {
      "uri": "scrape://store.example.com/products/SKU-4471",
      "name": "store.example.com — SKU-4471 (latest)",
      "mimeType": "application/json",
      "description": "Single product snapshot for SKU-4471. Last change detected 2026-01-30T09:15:02Z (price 49.99 → 44.99)."
    }
  ]
}

That description field matters more than it looks. It’s the only thing a cheap model reads before deciding whether to fetch the resource. Put the schema summary, the snapshot timestamp, and the refresh cadence in it. An agent that knows the snapshot is 10 minutes old can decide whether that’s acceptable instead of blindly re-scraping.

Implementing resources/list and resources/read Handlers

I’ll use the Python MCP SDK’s low-level Server class, because it gives direct access to notification sending, which we need later. The store backing these handlers is a SQLite file the scraper writes to — reads never touch the network.

import json
import sqlite3
from mcp.server import Server
from mcp.server.session import ServerSession
from mcp.types import (
    Resource, TextResourceContents, Implementation, ProtocolVersion,
)
from mcp.server.stdio import stdio_server

DB_PATH = "snapshots.db"

def db():
    conn = sqlite3.connect(DB_PATH)
    conn.row_factory = sqlite3.Row
    return conn

server = Server(Implementation(name="scrape-resources", version="1.0.0"))

@server.list_resources()
async def list_resources() -> list[Resource]:
    rows = db().execute(
        """SELECT uri, name, description, scraped_at, stale
           FROM snapshots WHERE is_latest = 1"""
    ).fetchall()
    return [
        Resource(
            uri=r["uri"],
            name=r["name"],
            description=(
                f"{r['description']} Snapshot taken {r['scraped_at']}Z."
                + (" MARKED STALE: last scrape attempt failed." if r["stale"] else "")
            ),
            mimeType="application/json",
        )
        for r in rows
    ]

@server.read_resource()
async def read_resource(uri: str) -> list[TextResourceContents]:
    row = db().execute(
        "SELECT content, scraped_at, stale FROM snapshots WHERE uri = ? AND is_latest = 1",
        (uri,),
    ).fetchone()
    if row is None:
        raise ValueError(f"Unknown resource URI: {uri}")
    return [
        TextResourceContents(
            uri=uri,
            mimeType="application/json",
            text=row["content"],
        )
    ]

The read handler is a pure lookup: one indexed SQLite query, serialize, return. The ValueError for unknown URIs surfaces to the agent as a proper protocol error rather than a silent empty string — important, because agents are bad at noticing that a tool “succeeded” with garbage.

If you prefer the FastMCP decorator style, @mcp.resource("scrape://{host}/{path}") gets you the same result with less boilerplate. I stick with the low-level server here because subscriptions require poking at the session object anyway.

One opinionated choice: I return application/json and store the scrape output as structured JSON (via extract_rules), not raw HTML. Raw HTML as a resource works, but it floods the agent’s context window. If you must serve prose, serve markdown — see scraping pages as markdown for LLM consumption — and trim before it hits the context.

Persisting Snapshots So Reads Never Trigger a Scrape

The invariant of this whole design: resources/read must never, under any circumstance, fetch anything. The moment a read can trigger a scrape, you’ve rebuilt the tool-call pattern with extra steps and a latency cliff. All fetching lives in the scraper, which runs on its own schedule.

The scraper fetches, diffs against the previous snapshot, and writes only when the content actually changed:

import hashlib, json, time, requests

def fetch_products():
    resp = requests.post(
        "https://api.finedata.ai/api/v1/scrape",
        headers={"Authorization": "Bearer fd_your_api_key"},
        json={
            "url": "https://store.example.com/products",
            "formats": ["markdown"],
            "extract_rules": {
                "products": {
                    "selector": ".product-card",
                    "fields": {
                        "sku": "[data-sku]",
                        "title": "h2",
                        "price": ".price",
                        "availability": ".stock-badge",
                    },
                }
            },
            "only_main_content": True,
            "use_residential": True,
            "max_retries": 3,
        },
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json()

def save_snapshot(uri: str, name: str, data: dict) -> bool:
    content = json.dumps(data, sort_keys=True)
    content_hash = hashlib.sha256(content.encode()).hexdigest()
    conn = db()
    prev = conn.execute(
        "SELECT content_hash FROM snapshots WHERE uri = ? AND is_latest = 1",
        (uri,),
    ).fetchone()

    if prev and prev["content_hash"] == content_hash:
        # No change. Touch the timestamp so staleness tracking stays honest.
        conn.execute(
            "UPDATE snapshots SET scraped_at = ? WHERE uri = ? AND is_latest = 1",
            (time.strftime("%Y-%m-%dT%H:%M:%S"), uri),
        )
        conn.commit()
        return False

    conn.execute("UPDATE snapshots SET is_latest = 0 WHERE uri = ?", (uri,))
    conn.execute(
        """INSERT INTO snapshots
           (uri, name, description, content, scraped_at, content_hash, is_latest, stale)
           VALUES (?, ?, ?, ?, ?, ?, 1, 0)""",
        (uri, name, "Product listing snapshot.", content,
         time.strftime("%Y-%m-%dT%H:%M:%S"), content_hash),
    )
    conn.commit()
    return True  # changed → caller emits a notification

The schema it writes to:

CREATE TABLE snapshots (
    id           INTEGER PRIMARY KEY AUTOINCREMENT,
    uri          TEXT NOT NULL,
    name         TEXT NOT NULL,
    description  TEXT NOT NULL DEFAULT '',
    content      TEXT NOT NULL,          -- JSON payload served to agents
    scraped_at   TEXT NOT NULL,          -- ISO 8601
    content_hash TEXT NOT NULL,          -- sha256 of content, for change detection
    is_latest    INTEGER NOT NULL DEFAULT 0,
    stale        INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_snapshots_uri_latest ON snapshots(uri, is_latest);

The hash-based diff is the piece that makes subscriptions meaningful. Without it you’d notify on every cron tick and agents would learn to ignore your notifications — the server that cried wolf. Diff first, notify only on real change. This is the same logic I described in re-scrape vs change detection, just applied at the MCP boundary.

Notifying Agents with resources subscribe and Updated Events

Now the part that makes this more than a read-only cache. When a subscribed resource changes, the server pushes notifications/resources/updated to every subscribed session. The scraper and the MCP server live in the same process here, so the save step just calls into the server’s subscriber registry:

subscribers: dict[str, set[ServerSession]] = {}  # uri -> sessions

@server.subscribe_resource()  # newer SDKs expose this; older ones route via
async def subscribe(uri: str) -> None:            # a custom notification handler
    session = server.request_context.session
    subscribers.setdefault(uri, set()).add(session)

async def notify_updated(uri: str):
    payload = {
        "method": "notifications/resources/updated",
        "params": {"uri": uri},
    }
    dead = []
    for session in subscribers.get(uri, set()):
        try:
            await session.send_resource_updated(uri)  # SDK helper
        except Exception:
            dead.append(session)
    for s in dead:
        subscribers[uri].discard(s)

And the scraper’s main loop ties it together:

async def scrape_loop():
    while True:
        data = fetch_products()
        changed = save_snapshot(
            "scrape://store.example.com/products/latest",
            "store.example.com — product listing (latest)",
            data,
        )
        if changed:
            await notify_updated("scrape://store.example.com/products/latest")
        await asyncio.sleep(1800)  # every 30 minutes

On the client side, an agent subscribes once and reacts to pushes:

async def watch_products(session):
    await session.subscribe_resource("scrape://store.example.com/products/latest")

    async for message in session.incoming_messages():
        if (message.method == "notifications/resources/updated"
                and message.params.uri == "scrape://store.example.com/products/latest"):
            result = await session.read_resource(message.params.uri)
            snapshot = json.loads(result.contents[0].text)
            print(f"[update] {len(snapshot['products'])} products, "
                  f"taken {result.contents[0].text[:0] or 'just now'}")

The key behavioral shift: the agent stops deciding when to fetch and starts reacting to what changed. That’s fewer wasted calls and, more importantly, fewer wasted context tokens — the agent only re-reads the resource when there’s a reason to.

A caveat worth stating plainly: subscriptions are the least mature corner of MCP. Client support varies; some clients subscribe and ignore notifications, some don’t subscribe at all. Design for it anyway — the fallback (agent re-reads the URI, gets the cached snapshot in milliseconds) degrades gracefully. If you’re wiring agents to web data more broadly, the patterns in MCP protocol and AI web data cover the surrounding terrain.

Verifying Freshness: Reading the Snapshot Without Re-Fetching

Claims about avoided traffic are cheap, so measure it. I ran a local harness: 10 agent data accesses against the same product data, once via a re-scraping tool, once via resource reads after the initial snapshot was in place.

	Tool call (re-scrape each time)	Resource read
HTTP requests to store.example.com	10	0 (after initial scrape)
Total wall time	~4.2s per access, ~42s total	~30ms per access, ~0.3s total
Scrape API calls billed	10	1 (the scheduled scrape)
Data freshness	Fresh at each call	Bounded by 30-min scrape cadence

That freshness trade-off is real and I’m not going to pretend otherwise. If your agent genuinely needs second-by-second freshness, resources are the wrong primitive — use a tool call and pay for it. For competitive monitoring, price tracking, and research tasks, a 30-minute-old snapshot is almost always fine, and the agent can see the snapshot age in the resource description and escalate if it isn’t.

An inspector session against the running server looks like this:

$ npx @modelcontextprotocol/inspector python server.py

▸ resources/list
  → 2 resources:
    scrape://store.example.com/products/latest
      description: "Product listing snapshot. Snapshot taken 2026-01-30T09:15:02Z..."
    scrape://store.example.com/products/SKU-4471
      description: "Single product snapshot for SKU-4471..."

▸ resources/read  scrape://store.example.com/products/latest
  → application/json, 14,832 bytes
    {"products": [{"sku": "SKU-4471", "title": "...", "price": "49.99", ...

▸ (scraper detects price change 49.99 → 44.99, saves snapshot, emits update)

  ← notification: resources/updated
       uri: scrape://store.example.com/products/latest

▸ resources/read  scrape://store.example.com/products/latest
  → application/json, 14,836 bytes
    {"products": [{"sku": "SKU-4471", "title": "...", "price": "44.99", ...

The notification arrived without the client polling anything, and the subsequent read reflected the new price. Zero requests hit the target site during the whole session.

## Handling Stale Data, Scraping Failures, and Resource Cleanup

Scrapes fail. Rate limits, redesigns, temporary outages. The rule I follow: a failed scrape never deletes or blanks the last good snapshot. The agent gets served slightly old data, clearly labeled, and can decide whether that’s acceptable. Silent staleness is worse than labeled staleness — an agent that thinks it’s reading current data will reason from it confidently and be wrong.

Extend the save step with an explicit failure path:

def mark_stale(uri: str, reason: str):
    conn = db()
    conn.execute(
        "UPDATE snapshots SET stale = 1, description = ? WHERE uri = ? AND is_latest = 1",
        (f"Last scrape attempt failed: {reason}. Serving previous snapshot.", uri),
    )
    conn.commit()

async def scrape_loop():
    while True:
        uri = "scrape://store.example.com/products/latest"
        try:
            data = fetch_products()
        except Exception as exc:
            mark_stale(uri, str(exc))
            await asyncio.sleep(300)  # retry sooner than the normal cadence
            continue
        # ... save_snapshot / notify_updated as before

The read handler already surfaces this: list_resources appends “MARKED STALE” to the description, and you can do the same inside read_resource by returning the stale flag in an envelope around the content. Some teams also emit notifications/resources/updated on the stale transition so agents hear about degradation, not just about changes. I’d do that only for prolonged outages — a single failed retry isn’t worth interrupting an agent mid-task.

Retention is the other operational detail. Every changed snapshot inserts a new row and flips the previous one to is_latest = 0. Left alone, that table grows forever. Keep a bounded history — 5 versions per URI is plenty for debugging and rollback — and purge the rest:

RETENTION_PER_URI = 5

def prune():
    conn = db()
    uris = conn.execute("SELECT DISTINCT uri FROM snapshots").fetchall()
    for u in uris:
        conn.execute(
            """DELETE FROM snapshots WHERE uri = ? AND is_latest = 0 AND id NOT IN (
                   SELECT id FROM snapshots WHERE uri = ?
                   AND is_latest = 0 ORDER BY id DESC LIMIT ?
               )""",
            (u["uri"], u["uri"], RETENTION_PER_URI),
        )
    conn.commit()

Run the prune after each save. If you also want age-based expiry — say, drop URIs nobody has scraped in 30 days — add a DELETE WHERE scraped_at < date('now', '-30 days') and make the scraper’s schedule authoritative over which URIs exist at all. A resource that no longer has a live scrape schedule should disappear from resources/list, not linger as a fossil.

Gotchas and Next Steps

Things that will bite you:

• Don’t put scrapes inside read handlers. Ever. The temptation is real (“just one quick fetch if the snapshot is old”) and it reintroduces the exact latency and cost problem you solved. Escalation belongs in the agent, or in the scrape schedule.
• Subscriptions are best-effort. Sessions die, clients ignore notifications, transports drop. Your correctness backstop is always the read — it must return the latest snapshot regardless of whether notifications worked.
• URI stability is a contract. If you rename scrape://store.example.com/products/latest to something else, every agent prompt referencing the old URI breaks silently. Version the scheme (scrape-v2://) rather than mutating meaning under a live address.
• Watch payload size. A 15KB JSON resource is fine; a 400KB one will eat an agent’s context. Trim at save time, not read time — trimming scraped pages before they flood context covers the mechanics.

Where to take it next: move the scraper to async jobs with a webhook callback so one process can maintain hundreds of URIs cheaply — the async scraping jobs and webhooks pattern drops straight into the save step above. You can also add a scrape:// directory resource per domain so agents can discover newly tracked pages without a server redeploy.

The mental model to keep: tools are for doing, resources are for knowing. Scraped snapshots are knowledge the server maintains on its own schedule. Serve them the way MCP intended — addressed, cached, and pushed when they change — and your scrape bill stops scaling with how talkative your agents are.