Test Your MCP Scraping Server Before an Agent Uses It

Why an MCP Server Needs a Test Harness Before an Agent Calls It

An agent discovering a broken MCP tool is worse than an agent finding no tool at all. When the tool is missing, the model says “I can’t do that” and the user moves on. When the tool exists but its schema lies, its responses are malformed, or its descriptions are empty, the agent will happily call it, misread the output, and confidently report wrong data. The failure is silent. Nobody checks. When you eventually do go looking, Debug MCP Tool Calls: Trace Agent Fetches shows how to reconstruct what the agent actually sent and received.

Scraping tools make this worse than average. A scrape_page tool depends on a live network, a third-party API, HTML that changes without notice, and a response contract that mixes content blocks with metadata. Any of those can drift between the day you wrote the server and the day an agent actually calls it. And the agent won’t tell you it broke — it will improvise. For a complementary pattern on handling targets that degrade over time, see Circuit Breaker: Catching Scrape Target Degradation.

Here is the mapping I care about. Each row is a stage where things go wrong, what the agent experiences, and what a local harness catches before any of that happens:

Failure stage	What the agent sees	What the harness catches locally
Tool discovery	Tool missing from list_tools, or present with an empty description so the model never selects it	Registry diff: expected tool names, non-empty descriptions, required properties present
Argument validation	Model invents a selector argument the schema never declared, or passes url as a number; server returns a 500 instead of a clean validation error	Schema round-trip: valid inputs accepted, invalid inputs rejected with a readable message
Response parsing	Tool returns {"html": ...} one day and {"content": [{"text": ...}]} the next; agent extracts from the wrong key and returns empty data	Contract check: response validated against a fixed schema every run
Timeout behavior	Tool hangs for 300 seconds, agent’s context window fills with retries or the whole session stalls	Latency budget and forced-timeout test with a hard ceiling
Pagination logic	Agent scrapes page 1, gets 20 items, declares the job done	Multi-page fixture run asserting item counts and next-page detection

The response-shape row deserves a concrete example. This is the kind of tool output that looks fine if you glance at it and quietly poisons an agent:

{
  "status": "ok",
  "data": {
    "title": "Example Domain",
    "items": null
  },
  "content": [
    {"type": "text", "text": "Example Domain"}
  ]
}

Two contracts in one response. data.items is null even though status says ok. The content block duplicates the title but drops everything else. An agent will pick whichever key looks plausible and move on. A harness rejects this before deployment because items must be an array and the payload must not carry two competing shapes.

If you want the background on why agents trip over these failure modes in practice, read MCP Tools for AI Agents: Auth Scopes and Failure Modes. The rest of this post is the fix: a local harness that exercises the server exactly the way an agent would, without an agent in the loop.

Scaffolding a Local MCP Client for Testing

You do not need a running LLM to test an MCP server. The protocol is just JSON-RPC over a transport, and the official SDKs give you a client you can drive from a test file. I use the Python SDK with stdio transport because it launches the server as a subprocess, which means the harness is self-contained: one command, no ports to manage, no leftover processes.

Install the client library alongside your test dependencies:

pip install mcp pytest

Then scaffold the session. This is the fixture every test reuses:

# conftest.py
import pytest
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

SERVER_CMD = ["python", "-m", "scraping_server"]

@pytest.fixture(scope="session")
async def mcp_session():
    params = StdioServerParameters(
        command=SERVER_CMD[0],
        args=SERVER_CMD[1:],
        env={
            "FINEDATA_API_KEY": "fd_your_api_key",
            "MCP_TRANSPORT": "stdio",
        },
    )
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            yield session

The connection settings live in a small config file so the harness and any manual inspection runs share them:

{
  "mcp_server": {
    "command": "python -m scraping_server",
    "transport": "stdio",
    "env": {
      "FINEDATA_API_KEY": "fd_your_api_key"
    }
  },
  "test_targets": {
    "simple": "https://example.com",
    "paginated": "https://store.example.com/products?page={n}",
    "slow": "https://api.example.com/delay/300"
  }
}

One opinionated choice here: I run the harness against the real network, not mocked HTTP responses. Mocks verify that your code matches your mock, which is nearly useless for a scraping tool whose entire job is talking to a real API. The fixture targets (example.com, store.example.com) are cheap, stable, and exercise the full path from tool call to upstream request and back. If a test needs the upstream scraping API, the server forwards to https://api.finedata.ai with the key from the environment. You can add a mock tier for CI if egress is a problem, but keep at least one scheduled run that hits the real thing.

Verifying Tool Discovery with list_tools

First test: the server actually exposes the tools you think it exposes. This sounds trivial. It is not — I have seen servers where a renamed function silently dropped a tool from the registry, and the agent just stopped offering that capability to users with no error anywhere.

# test_discovery.py

EXPECTED_TOOLS = {
    "scrape_page",
    "extract_links",
    "scrape_batch",
}

async def test_tool_registry(mcp_session):
    response = await mcp_session.list_tools()
    tools = {t.name: t for t in response.tools}

    print("Registered tools:")
    for name in sorted(tools):
        desc = (tools[name].description or "").strip()
        print(f"  {name}: {desc[:60]}")

    missing = EXPECTED_TOOLS - set(tools)
    assert not missing, f"Missing tools: {missing}"

    for name in EXPECTED_TOOLS:
        desc = (tools[name].description or "").strip()
        assert len(desc) >= 20, (
            f"Tool '{name}' has no usable description; "
            "the agent will never select it"
        )
        assert tools[name].inputSchema is not None

The description assertion matters more than people expect. Tool selection in an agent is driven almost entirely by the name and description the model sees in the registry. A tool registered as scrape_page with description "scrapes a page" competes poorly against ambiguity — the model will guess, or worse, pick a different tool. Treat the description as part of the tool’s public contract and test it like one.

I also assert inputSchema is present. A tool with a null schema accepts anything, which pushes validation failures downstream to where they are hardest to diagnose.

Asserting Argument Schemas Against Real Agent Inputs

Discovery passing tells you the tools exist. Next, confirm the schemas behave. Two directions matter: valid agent-shaped input succeeds, and invalid input fails with a message an agent can actually use.

The valid path, using a real call through the harness:

# test_arguments.py

async def test_scrape_page_valid_args(mcp_session):
    result = await mcp_session.call_tool("scrape_page", {
        "url": "https://example.com",
        "extract_rules": {"title": "h1", "body": "p"},
        "formats": ["markdown"],
        "timeout": 60,
    })
    assert not result.isError, f"Valid call failed: {result.content}"
    assert result.content, "No content blocks returned"

Under the hood the server forwards this to the scraping API — POST https://api.finedata.ai/scrape with the url, extract_rules, formats, and timeout fields in the body, authenticated with Authorization: Bearer fd_your_api_key. The harness does not care about that detail. It cares that the tool accepted the arguments an agent would plausibly send.

The invalid path is where most homegrown servers fall apart:

async def test_scrape_page_missing_url(mcp_session):
    result = await mcp_session.call_tool("scrape_page", {
        "extract_rules": {"title": "h1"},
    })
    assert result.isError, "Missing url should be rejected"
    text = result.content[0].text
    assert "url" in text.lower(), (
        f"Error message does not name the missing field: {text}"
    )

async def test_scrape_page_malformed_selector(mcp_session):
    result = await mcp_session.call_tool("scrape_page", {
        "url": "https://example.com",
        "extract_rules": {"title": "h1[[["},
    })
    assert result.isError, "Malformed selector should be rejected"
    text = result.content[0].text
    assert "selector" in text.lower() or "extract_rules" in text.lower(), (
        f"Error message is not actionable: {text}"
    )

The assertion on error message content is deliberate. An agent that receives "Internal error" from a tool cannot recover — it will retry the same call or abandon the task. An agent that receives "extract_rules.title: invalid CSS selector 'h1[[['" can fix the argument and try again. Validation errors are part of your tool’s UX. Test them like features.

A note on strictness: I prefer rejecting unknown arguments outright rather than silently ignoring them. If the model sends url_depth=2 and your server drops it, the agent believes it constrained the crawl when it did not. Loud failures beat silent no-ops. Some people disagree and prefer lenient servers; I think that trade only works when a human reads every response, which is not the situation agents create.

Validating Response Shape and Content Structure

Now the contract. Define, in code, exactly what a successful tool response looks like, and refuse to deploy anything that deviates. Pydantic works well here because the same model can validate in the harness and document the shape in the server’s README:

# contracts.py
from typing import Literal, Optional
from pydantic import BaseModel, Field

class ContentBlock(BaseModel):
    type: Literal["text"] = "text"
    text: str = Field(min_length=1)

class ScrapeMetadata(BaseModel):
    url: str
    status: int
    latency_ms: int
    format: str
    page: Optional[int] = None
    next_page: Optional[str] = None

class ScrapeResponse(BaseModel):
    status: Literal["ok", "error"]
    content: list[ContentBlock]
    data: Optional[dict] = None
    metadata: ScrapeMetadata

The test is then one line of substance:

# test_contract.py
from contracts import ScrapeResponse

async def test_scrape_page_response_contract(mcp_session):
    result = await mcp_session.call_tool("scrape_page", {
        "url": "https://example.com",
        "formats": ["markdown"],
    })
    assert not result.isError
    payload = json.loads(result.content[0].text)
    ScrapeResponse.model_validate(payload)  # raises on any deviation

A conforming response from https://example.com looks like this:

{
  "status": "ok",
  "content": [
    {
      "type": "text",
      "text": "# Example Domain\n\nThis domain is for use in illustrative examples in documents."
    }
  ],
  "data": {
    "title": "Example Domain"
  },
  "metadata": {
    "url": "https://example.com",
    "status": 200,
    "latency_ms": 812,
    "format": "markdown",
    "page": null,
    "next_page": null
  }
}

One contract, one shape. The data key holds structured extraction output when rules or a schema were supplied; content holds the human-readable block the agent can quote directly. The malformed example from the first section fails this model twice — data.items as null instead of an array, and no metadata block at all.

If your tool returns markdown for agent consumption, the format choice itself matters. For the case for that over dumping raw HTML into the response, read Stop Cleaning HTML by Hand: Scrape Pages as Markdown. The harness angle: whatever format you pick, freeze it in the contract model so a refactor cannot silently change it.

Testing Pagination and Multi-Page Scrapes Against store.example.com

Single-page tests catch obvious breakage. Pagination tests catch the logic bugs — off-by-one page handling, inconsistent item counts, and the classic: next-page detection that works on page 1 and fails on the last page because the “next” link disappears.

Run the fixture site’s product listing through the tool, pages 1 through 3:

# test_pagination.py

async def collect_page(mcp_session, page: int):
    result = await mcp_session.call_tool("scrape_page", {
        "url": f"https://store.example.com/products?page={page}",
        "extract_rules": {
            "items": ".product-card",
            "item_name": ".product-card h3",
            "price": ".product-card .price",
        },
        "formats": ["markdown"],
    })
    assert not result.isError, f"Page {page} failed: {result.content}"
    return json.loads(result.content[0].text)

async def test_pagination_consistency(mcp_session):
    pages = [await collect_page(mcp_session, n) for n in (1, 2, 3)]

    counts = [len(p["data"]["items"]) for p in pages]
    assert counts == [20, 20, 20], f"Item counts drifted: {counts}"

    # next_page present on 1 and 2, absent on the last page
    assert pages[0]["metadata"]["next_page"] == \
        "https://store.example.com/products?page=2"
    assert pages[1]["metadata"]["next_page"] == \
        "https://store.example.com/products?page=3"
    assert pages[2]["metadata"]["next_page"] is None, \
        "Last page still advertises a next page"

    # every item actually filled its fields
    for p in pages:
        for item in p["data"]["items"]:
            assert item["item_name"], "Empty item_name in extraction"
            assert item["price"], "Empty price in extraction"

The output comparison across the three calls should look like this, and if it does not, that diff is your bug report:

page=1  items=20  next_page=/products?page=2  latency_ms=940
page=2  items=20  next_page=/products?page=3  latency_ms=880
page=3  items=20  next_page=None              latency_ms=910

The empty-field assertions are the part people skip. An extraction rule that matches zero nodes on page 2 but matches fine on page 1 produces a “successful” scrape with hollow data. Agents do not notice. Your harness has to. For a fuller treatment of measuring extraction quality, read Audit Field Fill Rates: Measure Extraction Quality Locally, which goes deep on per-field accounting.

Measuring Latency and Timeout Behavior Locally

Agents have patience your users do not. A tool that takes 90 seconds will get called, and the agent will wait, and the user will stare at a spinner. So the harness records latency per call and fails the build when it drifts past a budget you set deliberately.

# test_latency.py
import time, statistics

async def measure(mcp_session, url: str, n: int = 5):
    samples = []
    for _ in range(n):
        t0 = time.monotonic()
        result = await mcp_session.call_tool("scrape_page", {
            "url": url,
            "formats": ["markdown"],
            "timeout": 60,
        })
        dt = (time.monotonic() - t0) * 1000
        assert not result.isError
        samples.append(dt)
    return samples

async def test_latency_budget(mcp_session):
    samples = await measure(mcp_session, "https://example.com")
    median = statistics.median(samples)
    p95 = sorted(samples)[int(len(samples) * 0.95) - 1]
    print(f"median={median:.0f}ms p95={p95:.0f}ms samples={samples}")
    assert p95 < 5000, f"p95 latency {p95:.0f}ms exceeds 5s budget"

Typical output from a healthy run:

Target	Calls	Median	p95	Budget
https://example.com (markdown)	5	870 ms	1,240 ms	5,000 ms
https://example.com (markdown + rules)	5	910 ms	1,380 ms	5,000 ms

Five samples will not win you a statistics prize. It is enough to catch a regression from 900 ms to 9 seconds, which is the class of change that actually hurts agents. Tune the sample count to your CI tolerance.

Then the timeout test. Point the tool at a deliberately slow endpoint — something that sleeps longer than the tool’s timeout — and assert the failure is fast and legible rather than a hang:

async def test_timeout_returns_clean_error(mcp_session):
    t0 = time.monotonic()
    result = await mcp_session.call_tool("scrape_page", {
        "url": "https://api.example.com/delay/300",
        "formats": ["markdown"],
        "timeout": 10,
    })
    elapsed = time.monotonic() - t0
    assert result.isError, "Slow target should produce an error"
    assert elapsed < 20, (
        f"Tool took {elapsed:.0f}s to fail; timeout is not enforced"
    )
    text = result.content[0].text
    assert "timeout" in text.lower(), f"Unhelpful error: {text}"

The elapsed-time assertion is the real test. Plenty of servers eventually return a timeout error — after the underlying client’s own 300-second ceiling, which is 30x your budget. Enforce the wall-clock bound, not just the error type. And when you do retry timed-out scrapes, remember that retries have a cost signature. For the accounting, read How Retries Quietly Double Your Scraping Costs.

Wiring the Harness into CI as a Pre-Agent Gate

Everything above is a pytest suite. Making it a gate means running it on every commit and treating a failure as a deployment blocker, not a warning. GitHub Actions version:

# .github/workflows/mcp-harness.yml
name: mcp-harness
on: [push, pull_request]

jobs:
  harness:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install mcp pytest pydantic
      - run: pytest tests/mcp/ -v --tb=short
        env:
          FINEDATA_API_KEY: ${{ secrets.FINEDATA_API_KEY }}

Two details worth flagging. The job-level timeout-minutes: 10 is a backstop — if the harness itself hangs on a stuck stdio session, CI kills it instead of burning runner minutes. And the API key comes from secrets, never from the config file, which stays committed with the placeholder value.

When the harness catches a regression, the log reads like this:

tests/mcp/test_contract.py::test_scrape_page_response_contract FAILED

E   pydantic_core.ValidationError: 2 validation errors for ScrapeResponse
E   data.items
E     Input should be a valid list [type=list_type, input_value=None]
E   metadata
E     Field required [type=missing]

That is a schema mismatch caught in CI, roughly thirty seconds after the commit that caused it. The alternative timeline: the change ships, an agent calls scrape_page next week, reads data.items as null, tells the user the store has no products, and nobody files a bug because the answer looked plausible. I know which timeline I want.

One honest trade-off to close on: this setup makes CI dependent on external network calls, which means occasional flaky runs when a fixture site or the upstream API hiccups. Mitigate with retries inside the fixtures and a scheduled nightly run that tolerates no flakes. Do not “solve” it by deleting the network tests — a scraping server verified only against mocks is a server verified against nothing.

Wrap-Up

The pattern is small enough to hold in your head: launch the server over stdio, assert the tool registry, exercise valid and invalid arguments, validate every response against a frozen contract, run pagination fixtures, enforce latency and timeout budgets, and gate deploys on all of it. Maybe 300 lines of test code for a typical scraping server.

Gotchas I hit repeatedly:

• Session-scoped fixtures leak state. If your server caches anything between tool calls, use a fresh session per test or you will chase phantom failures.
• Error text assertions are brittle to rewording. Assert on field names appearing in the message, not exact phrasing.
• example.com is fine for contract tests but too static for extraction tests. You need a fixture with real structure — store.example.com or a locally served HTML file — or your selector assertions prove nothing.
• Latency budgets need periodic recalibration. A budget set once and forgotten either rots into meaninglessness or starts failing randomly when upstream latency shifts.

Next steps, in order of payoff: add a contract test for every tool, not just scrape_page — extract_links and batch tools deserve the same treatment. Then add a change-detection test that stores the last-seen extraction output from each fixture and diffs it, so you learn about upstream HTML changes from CI rather than from a user. And if your agents consume these tools through cached layers, Cache MCP Tool Responses: Stop Re-Fetching the Same Page pairs well with this harness, because a cache in front of an unvalidated tool just makes wrong answers faster.

Test the tools the way the agent will use them: blind, trusting, and never reading your source code.