MCP Protocol: How to Connect AI Agents to Web Data

MCP Protocol: How to Connect AI Agents to Web Data

Large language models are remarkably capable at understanding and generating text, but they have a fundamental limitation: they can only work with information they have been trained on or that is provided in their context window. For tasks that require current web data — prices, product listings, documentation, news, regulatory filings — LLMs need a bridge to the live internet.

The Model Context Protocol (MCP) is that bridge. Developed by Anthropic as an open standard, MCP provides a structured way for AI agents and LLM-powered applications to interact with external tools and data sources. Below: what MCP is, why it matters for web data access, and how to set up FineData’s MCP server for use with Cursor IDE and Claude Desktop.

Update (2026-08): the server described here is now v0.3.1. It runs locally over stdio or remotely over Streamable HTTP at https://mcp.finedata.ai/mcp, where a client signs in with OAuth 2.1 rather than carrying a key. Nine tools — reading and writing are separate ones since 0.3.0, so scrape_url is always a GET — and an escalation ladder that pairs premium rendering with ISP routing before reaching for residential exits. Connection details for every client live on the MCP server page.

What is the Model Context Protocol?

MCP is an open protocol that standardizes how AI applications communicate with external tools and data sources. Think of it as a USB-C port for AI — a universal connection standard that allows any compliant AI client to work with any compliant tool server.

The protocol defines three primitives:

• Tools — Functions that the AI can call to perform actions (e.g., scrape a URL, search a database)
• Resources — Data sources that the AI can read (e.g., files, database records, API responses)
• Prompts — Pre-defined prompt templates that help the AI use tools effectively

The architecture follows a client-server model:

AI Application (Client)
    ↓ MCP Protocol (JSON-RPC over stdio or Streamable HTTP)
MCP Server
    ↓ Internal APIs
External Services (web scraping, databases, APIs)

An AI application (like Cursor or Claude Desktop) acts as the MCP client. It discovers available tools from connected MCP servers and can invoke them when the user’s request requires external data.

Why AI Agents Need Web Data Access

Consider these common scenarios where an LLM needs live web information:

Research and analysis. “Summarize the latest pricing changes on competitor X’s website.” The LLM needs to fetch and read the current state of a webpage.

Code generation with current docs. “Write an integration using the latest Stripe API.” Library documentation changes frequently — the LLM’s training data may reference outdated endpoints or deprecated parameters.

Data extraction. “Extract all product listings from this category page and format them as CSV.” The LLM can understand the task semantically but needs access to the actual page content.

Monitoring and alerting. “Check if this government regulation page has been updated since last week.” Requires fetching and comparing web content over time.

Content aggregation. “Gather the top 10 results for this search query and synthesize the key findings.” Requires scraping multiple pages and combining the information.

Without MCP (or a similar tool-use protocol), these tasks require manual copy-pasting of web content into the AI’s context — a tedious, error-prone process that does not scale.

A Scraping API Exposed as an MCP Server

The server package (finedata-mcp on PyPI, @finedata/mcp-server on npm) exposes web scraping capabilities as MCP tools. Once connected, the model can fetch and process live webpages with full-browser rendering, matching TLS profiles, captcha solving and proxy management behind a single tool call. Pages come back as markdown by default, because raw HTML burns context the model has to pay for, and a request that never rendered a page is not billed.

Available Tools

Nine tools, split into the work itself and the bookkeeping an autonomous agent needs to stay oriented:

Tool	Description
scrape_url	Read one URL with GET and return markdown, HTML, links or structured JSON
send_http_request	Send POST, PUT, PATCH or DELETE through the same pipeline — forms, write APIs
batch_scrape	Submit up to 100 URLs in one call with shared options
get_batch_status	Track a batch and collect per-URL results
scrape_async	Queue a long-running scrape and get a job id back
get_job_status	Poll one queued job and read its result when finished
list_jobs	List recent jobs with their state, to resume after a restart
cancel_job	Stop a queued job that is no longer needed
get_usage	Read token balance and consumption, so the agent can budget itself

Each tool accepts the same parameters as the REST API, giving the model full control over scraping configuration:

{
  "url": "https://example.com/product/1",
  "formats": ["markdown"],
  "only_main_content": true,
  "stealth_premium": true,
  "use_isp": true
}

Escalating by Cost Instead of by Habit

An agent that reaches for the heaviest option on every request is expensive; one that never escalates returns empty pages. The tool descriptions therefore carry per-mode costs, and the ladder is ordered by what actually works. Start plain — TLS fingerprint matching alone (tls_profile defaults to a current Chrome profile) clears ordinary sites for about three tokens. If the page comes back blocked or empty, add stealth_antibot. The strongest common step is stealth_premium together with use_isp: on a benchmark of hard targets that pairing succeeded roughly six times out of ten, while the same premium rendering over datacenter IPs managed about one in eight. Residential and mobile exits sit above that, and they are for genuinely geo- or IP-sensitive targets rather than for routine escalation.

Every response reports tokens_used, and failed requests are not billed, so an agent can measure its own spend rather than infer it from the invoice later.

What Happens Under the Hood

When the AI decides to use the scrape_url tool:

1. The AI client sends a tools/call request to the MCP server via JSON-RPC
2. The MCP server constructs an API request to the scraping infrastructure
3. Servers handle browser rendering, TLS profile selection, proxy rotation, and content extraction
4. The scraped content is returned to the MCP server
5. The MCP server formats the response and sends it back to the AI client
6. The AI incorporates the web content into its context and generates a response

The entire process is transparent to the user — you ask the AI a question that requires web data, and it fetches what it needs automatically.

Setting Up with Cursor IDE

Cursor is an AI-powered code editor that supports MCP servers natively. Here is how to connect the MCP server:

Step 1: Decide Where the Server Runs

Nothing has to be installed ahead of time. uvx and npx fetch and run the package on demand, so the editor starts the server as a child process and speaks to it over stdio. If you would rather pin it, pip install finedata-mcp or npm i -g @finedata/mcp-server work the same way.

The alternative is not to run it at all: the hosted endpoint answers over Streamable HTTP, which suits agents on machines you do not control and avoids updating a local package.

Step 2: Configure Cursor

Open Cursor’s settings and navigate to the MCP configuration. Add the server under a clear name. In your Cursor MCP configuration file (typically ~/.cursor/mcp.json or in your project’s .cursor/mcp.json):

{
  "mcpServers": {
    "finedata": {
      "command": "uvx",
      "args": ["finedata-mcp"],
      "env": {
        "FINEDATA_API_KEY": "fd_your_api_key"
      }
    }
  }
}

For the hosted endpoint, the same file takes a URL instead of a command:

{
  "mcpServers": {
    "finedata": {
      "url": "https://mcp.finedata.ai/mcp",
      "headers": { "Authorization": "Bearer fd_your_api_key" }
    }
  }
}

Step 3: Verify the Connection

Open Cursor and start a new chat. Ask the AI to scrape a webpage:

“Fetch the homepage of https://example.com and summarize its content.”

The AI should invoke the scrape_url tool, fetch the page, and provide a summary. You can see the tool invocation in the chat interface.

Practical Use Cases in Cursor

Once connected, you can use web scraping naturally within your development workflow:

Checking current API documentation:

“Scrape the Stripe API documentation for webhooks and show me how to verify a webhook signature in Python.”

Competitive analysis:

“Fetch the pricing page at competitor.com/pricing and compare their plans to ours.”

Extracting test data:

“Scrape https://jsonplaceholder.typicode.com/posts and create a TypeScript interface that matches the response structure.”

Debugging integration issues:

“Fetch https://api.example.com/health and tell me if their service is currently operational.”

Setting Up with Claude Desktop

Claude Desktop also supports MCP servers, and the configuration is nearly identical — same package, same runner, a different file.

Step 1: Configure Claude Desktop

Edit Claude Desktop’s configuration file. On macOS, this is typically located at ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "finedata": {
      "command": "uvx",
      "args": ["finedata-mcp"],
      "env": {
        "FINEDATA_API_KEY": "fd_your_api_key"
      }
    }
  }
}

Step 2: Restart and Verify

Restart Claude Desktop. You should see the scraping tools available in the tools panel. Test with a simple scraping request:

“Scrape https://news.ycombinator.com and tell me the top 5 stories right now.”

Claude will call the scrape_url tool, retrieve the Hacker News front page, and summarize the current top stories.

Connecting Without Handing Over a Key

Both configurations above put an API key in a file. That is reasonable for an agent you run yourself and awkward for one you hand to a teammate or ship to a customer, because an API key is a bearer secret — whoever holds it is you.

The hosted endpoint therefore speaks OAuth 2.1, which changes the shape of the problem. A compliant client registers itself dynamically, so there is no client id to create by hand:

{
  "mcpServers": {
    "finedata": {
      "url": "https://mcp.finedata.ai/mcp"
    }
  }
}

With nothing else configured, the client discovers the endpoint’s metadata, opens a browser on a consent screen naming the scopes it wants, and receives a short-lived token verified with PKCE. Each person connects their own account and their own balance. Three scopes exist — scrape:write to fetch and cancel, jobs:read to poll, usage:read to check the balance — and the token reaches those tools and nothing else, so it cannot create API keys, change a plan or read profile data. It expires after an hour, refreshes for up to 30 days, and dies on a password change or a revoke from the dashboard.

Advanced Usage Patterns

Batch Research

When your AI agent needs to gather information from multiple sources:

“I need to compare pricing for cloud GPU instances. Scrape the pricing pages from AWS, Google Cloud, Azure, and Lambda Labs, then create a comparison table.”

The AI uses batch_scrape to fetch all four pages in parallel, then synthesizes the information into a structured comparison.

Iterative Data Extraction

For multi-page data extraction where results span multiple pages:

“Scrape this product category page. If there are pagination links, follow them and collect all product names and prices.”

The AI can make multiple sequential scrape_url calls, following pagination links found in each response.

Monitoring and Change Detection

Combine web scraping with the AI’s analytical capabilities:

“Fetch this competitor’s feature page. I scraped it last week and saved the content in features-last-week.md. What has changed?”

The AI scrapes the current version, compares it against the saved content, and highlights differences.

Data Pipeline Integration

For applications that need regular data feeds, the async scraping tools enable background processing:

# Example: AI agent triggering a background scrape job
# The AI calls scrape_async with a list of URLs
# Later, it checks get_job_status to retrieve results

Security Considerations

When connecting AI agents to web scraping capabilities, keep these security practices in mind:

API key management. Store your API key in environment variables, not in configuration files checked into version control. Use separate API keys for development and production. For an agent that someone else runs, prefer the OAuth connection described above: it scopes what the agent can reach and can be revoked without rotating a shared secret.

Rate limiting. While the API handles server-side rate limiting, be aware that AI agents can make rapid sequential requests. Set reasonable limits in your application logic to control costs. Failed renders are not billed, but rapid retries still consume time and capacity.

Content validation. The AI will incorporate scraped content into its responses. For sensitive applications, validate that the scraped content matches expected patterns before acting on it.

Data handling. Web scraping may return personal data or copyrighted content. Ensure your use of scraped data complies with applicable regulations. See our legal guide for more details.

The Future of AI + Web Data

MCP represents a fundamental shift in how AI applications interact with the outside world. Rather than relying solely on training data (which is inevitably stale), AI agents can access live, current information from any website.

As AI agents become more autonomous — planning multi-step research tasks, monitoring data sources, and maintaining up-to-date knowledge bases — the ability to reliably scrape web data becomes a core capability rather than an optional add-on.

The combination of MCP’s standardized protocol, a robust scraping infrastructure, and the reasoning capabilities of modern LLMs creates a powerful stack for building AI applications that are truly connected to the real world.

Summary

1. MCP standardizes AI-to-tool communication. It provides a universal protocol for AI applications to discover and use external tools.
2. Web scraping is a natural MCP use case. AI agents frequently need current web data that is not in their training set.
3. Setup is one config block. The same mcpServers entry works in Cursor, Claude Desktop and any other compliant client, either as a local process or as a URL pointing at the hosted endpoint.
4. The AI handles complexity. You describe what you need in natural language; the AI figures out how to scrape, parse, and present the data.
5. Cost belongs in the tool contract. Per-mode costs in the tool descriptions and tokens_used in every response let an agent escalate on evidence and stop itself before the budget does.
6. Security matters. Manage API keys carefully, prefer OAuth for agents you do not run yourself, and be mindful of data handling practices.

---

Ready to give your AI agent web access? The MCP server page has the config for every client, the full tool reference and the scopes an OAuth connection grants. Create an account to get free tokens and connect it in under five minutes.