Confirm Before You Fetch: MCP Elicitation for Scraping Tools
Add MCP elicitation to scraping tools: pause for user consent on costly fetches, collect credentials mid-run, and fail cleanly when no one answers.
On this page
Why Costly Fetches Need a Consent Gate: The Case for MCP Elicitation in Scrapers
An agent with a scraping tool will happily burn through your token budget while you’re getting coffee. It sees a URL, it calls the tool, it gets data. That’s the whole point — until the fetch in question costs 30 tokens per page because it needs JS rendering and a residential proxy, or until it hits a page that requires your logged-in session, or until it fires 200 requests at a target you’d rather approach gently.
The Model Context Protocol has a mechanism built for exactly this: elicitation. When the server sends an elicitation request, the client pauses, renders a form for the human operator, and returns the answer. The tool call effectively suspends until a person says yes, no, or fills in a field. For scraping tools, this turns “the agent decided to fetch” into “the agent asked, and you approved.”
Here’s the honest comparison of what changes:
| Dimension | Fire-and-forget fetch | Elicitation-gated fetch |
|---|---|---|
| Cost per call | Spent immediately, no veto | Spent only after approval |
| Side effects | Hits the target site unattended | Human sees the target before the hit |
| Credential handling | Must be pre-provisioned in config | Collected at the moment of use |
| Failure mode on ambiguity | Agent guesses and proceeds | Agent asks, or times out cleanly |
| User awareness | None until the invoice arrives | Explicit, per fetch or per domain |
| Latency | One round trip | Two round trips plus human reaction time |
The trade-off is real: you’re adding a human to the critical path. On a batch of 10,000 URLs, you do not want a confirmation per URL. On a one-off fetch that costs premium tokens or touches an authenticated session, you absolutely do. The rest of this post is about drawing that line correctly.
The round trip looks like this:
MCP Server MCP Client User
| | |
| tool call: | |
| scrape(url) | |
|<------------------| |
| | |
| elicitation: | |
| "approve fetch | |
| for $URL?" | |
|------------------>| render form |
| |----------------->|
| | | (user clicks approve
| |<-----------------| or declines)
| elic result | |
|<------------------| |
| | |
| [if approved] | |
| HTTP fetch | |
| to target | |
| | |
| tool result | |
|------------------>| |
If you’re already running MCP tools for agents, the failure modes of unattended fetches will look familiar — auth scopes that break silently are a close cousin. Elicitation is the structural fix for the subset of those failures that a human could have caught in two seconds.
Modeling Consent as an Elicitation Schema: Defining the Confirm-Before-Fetch Request
An elicitation request carries a requestedSchema — a JSON Schema subset the client renders as a form. Keep it small. Every field you add is friction, and the operator is often reading this on a small approval dialog, not a full page.
For a scraping consent gate, three fields matter: what you’re fetching, what it will cost, and the decision itself.
// consent-schema.ts
export const FetchConsentSchema = {
name: "fetch_consent",
title: "Approve costly fetch",
description:
"This scrape uses premium options. Approve to proceed, decline to cancel.",
schema: {
type: "object" as const,
properties: {
target_url: {
type: "string",
title: "Target URL",
description: "The page the tool will fetch",
},
estimated_cost: {
type: "string",
title: "Estimated cost",
description: "Token cost estimate for this fetch",
},
reason: {
type: "string",
title: "Why this fetch is gated",
description: "Trigger that caused the consent gate",
},
approve: {
type: "boolean",
title: "Approve this fetch?",
description: "True to proceed, false to cancel the tool call",
},
},
required: ["approve"],
},
} as const;
Note what’s deliberately absent: I’m not asking for target_url as an input. The server fills it in as a read-only display value; the only field the user actually interacts with is approve. If your client supports readOnly annotations on properties, mark the display fields read-only. If it doesn’t, the required list does the work — the form can’t be submitted without a decision.
The wire message the client receives looks like this:
{
"jsonrpc": "2.0",
"id": 42,
"method": "elicitation/create",
"params": {
"message": "Fetch https://example.com/products with JS rendering and residential proxy (~8 tokens)?",
"requestedSchema": {
"type": "object",
"properties": {
"target_url": {
"type": "string",
"title": "Target URL"
},
"estimated_cost": {
"type": "string",
"title": "Estimated cost"
},
"reason": {
"type": "string",
"title": "Why this fetch is gated"
},
"approve": {
"type": "boolean",
"title": "Approve this fetch?"
}
},
"required": ["approve"]
}
}
}
The message string is your one shot at context. Don’t waste it on “Confirm?” — say what will happen, at what cost, and why it triggered the gate. “Residential proxy required: example.com blocks datacenter IPs” is a decision the operator can actually make.
Wiring Elicitation into the Scraping Tool Handler: Pausing the Fetch Mid-Run
Now the handler. The tool receives a scrape request, decides whether it’s “costly” enough to gate, asks, and only fetches on approval. The fetch itself goes through the scrape API so the tool gets retries, proxy routing, and rendering in one call.
// tools/scrape.ts
import { z } from "zod";
import type { Server } from "@modelcontextprotocol/sdk/server/index.js";
const SCRAPE_API = "https://api.finedata.ai/api/v1/scrape";
const API_KEY = process.env.FINEDATA_API_KEY;
const CostlyScrapeInput = z.object({
url: z.string().url(),
use_js_render: z.boolean().optional(),
use_residential: z.boolean().optional(),
solve_captcha: z.boolean().optional(),
formats: z.array(z.string()).optional(),
});
// Anything above this token estimate needs a human yes
const COST_GATE_THRESHOLD = 5;
function estimateTokens(input: z.infer<typeof CostlyScrapeInput>): number {
let t = 1; // base request
if (input.use_js_render) t += 5;
if (input.use_residential) t += 3;
if (input.solve_captcha) t += 10;
return t;
}
export function registerScrapeTool(server: Server) {
server.registerTool("scrape_page", {
inputSchema: CostlyScrapeInput.shape,
description: "Fetch a page through the scraping API, with consent gating",
}, async ({ url, ...options }) => {
const cost = estimateTokens({ url, ...options });
if (cost > COST_GATE_THRESHOLD) {
const result = await server.elicit({
message: `Fetch ${url} (~${cost} tokens: JS render / residential / captcha)?`,
requestedSchema: FetchConsentSchema.schema,
});
// Server fills display fields; client returns only what the user set
if (result.action !== "accept" || result.content?.approve !== true) {
return {
content: [{
type: "text",
text: `Fetch declined by operator for ${url}. Nothing was requested.`,
}],
isError: true,
};
}
}
// Only reachable after approval (or when under the cost threshold)
const response = await fetch(SCRAPE_API, {
method: "POST",
headers: {
"Authorization": `Bearer ${API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ url, ...options }),
});
if (!response.ok) {
return {
content: [{ type: "text", text: `Scrape API returned ${response.status}` }],
isError: true,
};
}
const data = await response.json();
return {
content: [{ type: "text", text: JSON.stringify(data) }],
};
});
}
The key structural point: the fetch(SCRAPE_API, ...) call sits lexically and logically after the approval check. There is no code path where the HTTP request fires before the elicitation resolves. When the operator declines, the target site is never touched — the tool returns an error result to the model without a single outbound request. That property matters more than it looks. “We didn’t hit the site” and “we hit the site and discarded the result” are very different statements when someone asks why their domain saw traffic from your agent.
One opinion here: gate on estimated cost, not on URL count. A hundred cheap plain-GET fetches to example.com are usually less consequential than one headful stealth render with captcha solving. Cost-based gating also degrades gracefully — as your pricing model changes, you adjust one threshold instead of rewriting URL allowlists. That said, if you operate in a regulated context, keep a separate hard allowlist for domains that must never be fetched without consent regardless of cost. Cost is a proxy for impact, not a guarantee.
Collecting Credentials Mid-Run: Elicitation Login Secrets Without Storing Them
The second use case is more interesting. Some pages need an authenticated session — a store’s admin dashboard, a partner portal, a staging environment behind basic auth. The standard approach is to bake credentials into the MCP server config. That works, but it means the secret sits on disk indefinitely and is available to every tool call, attended or not.
Elicitation gives you an alternative: ask for the credential at the moment of use, hold it in memory for one request, and let it go.
// tools/auth-scrape.ts
const CredentialSchema = {
type: "object" as const,
properties: {
username: {
type: "string",
title: "Username",
description: "Account for store.example.com",
},
password: {
type: "string",
title: "Password",
description: "Passed to the fetch, never written to disk",
// Clients that honor this mask the input field
format: "password",
},
},
required: ["username", "password"],
} as const;
export async function scrapeAuthenticatedCatalog(server: Server) {
const cred = await server.elicit({
message:
"store.example.com/catalog requires a session. Provide credentials for this one fetch?",
requestedSchema: CredentialSchema,
});
if (cred.action !== "accept" || !cred.content?.password) {
return {
content: [{ type: "text", text: "Credentials not provided; fetch aborted." }],
isError: true,
};
}
// Scope the secret to this block so it's unreachable afterward
const headers = {
Authorization:
"Basic " +
Buffer.from(`${cred.content.username}:${cred.content.password}`).toString("base64"),
};
const response = await fetch("https://api.finedata.ai/api/v1/scrape", {
method: "POST",
headers: {
"Authorization": `Bearer ${process.env.FINEDATA_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
url: "https://store.example.com/catalog",
headers, // forwarded with the scrape request
use_js_render: true,
formats: ["markdown"],
}),
});
const data = await response.json();
// Drop references; nothing was persisted, nothing outlives this call
headers.Authorization = "";
return {
content: [{ type: "text", text: JSON.stringify(data) }],
};
}
Two things to be honest about. First, “not storing” is a best effort, not a guarantee — the credential transits through the MCP client, the elicitation response, and process memory. A determined local attacker can grab it. What you’re actually buying is a much smaller blast radius: no secret at rest, no secret available to unattended runs, no secret in your git history. Second, the format: "password" hint is advisory; clients may ignore it and render a plain text field. Test with your actual client before promising anyone masked input.
The pattern pairs well with sticky sessions when the login flow is multi-step — once you’ve authenticated, you can keep the same exit IP for follow-up fetches so the site doesn’t flag the session hopping between addresses.
Failing Cleanly When Nobody Answers: Timeouts, Cancellations, and Error Semantics
Elicitation introduces a failure mode your scraping tools never had before: the human might not be there. The operator closed the laptop. The client renders the form but nobody’s watching. If your handler awaits forever, the tool call hangs, the agent stalls, and eventually something upstream times out in a way that produces a confusing stack trace instead of an actionable message.
Wrap the elicitation in a timeout and return a structured error.
// lib/with-timeout.ts
export async function elicitWithTimeout(
server: Server,
params: ElicitRequest,
timeoutMs = 30_000
): Promise<ElicitResult> {
return Promise.race([
server.elicit(params),
new Promise<never>((_, reject) =>
setTimeout(
() => reject(new ConsentTimeoutError(timeoutMs)),
timeoutMs
)
),
]);
}
export class ConsentTimeoutError extends Error {
readonly code = "CONSENT_TIMEOUT";
constructor(readonly timeoutMs: number) {
super(`No operator response within ${timeoutMs}ms`);
}
}
Then, in the handler, map every non-approval outcome to the same error shape:
try {
const result = await elicitWithTimeout(server, {
message: `Approve fetch for ${url}?`,
requestedSchema: FetchConsentSchema.schema,
});
if (result.action === "reject" || result.action === "cancel") {
throw new ConsentDeclinedError(result.action);
}
if (result.content?.approve !== true) {
throw new ConsentDeclinedError("no_approval_field");
}
} catch (err) {
const reason =
err instanceof ConsentTimeoutError ? "consent_timeout"
: err instanceof ConsentDeclinedError ? "consent_declined"
: "consent_error";
return {
isError: true,
content: [{
type: "text",
text: JSON.stringify({
error: "fetch_not_authorized",
reason,
target_url: url,
hint: reason === "consent_timeout"
? "No operator responded. Ask the user to confirm before retrying."
: "The operator declined this fetch. Do not retry without new instructions.",
}),
}],
};
}
The error the model sees:
{
"error": "fetch_not_authorized",
"reason": "consent_timeout",
"target_url": "https://example.com/products",
"hint": "No operator responded. Ask the user to confirm before retrying."
}
The hint field is doing real work. Models retry. If your error just says “failed,” a capable agent will try again, which triggers another elicitation, which times out again, and now you’ve built an infinite loop that pings the operator every 30 seconds. The hint must tell the model to surface the question to the user in conversation instead of retrying the tool. On consent_declined, be even more explicit: the operator said no, retrying is insubordination.
Distinguish the two outcomes in your telemetry too. A timeout means your UX is wrong or the operator stepped away. A decline means your agent is proposing fetches a human doesn’t want — that’s a prompt or policy problem, not a latency problem. Same error channel, very different fixes.
Testing the Consent Flow Locally: Measuring Latency and Verifying the Gate Holds
Before you ship this, verify two things: the gate actually holds (no fetch path bypasses approval), and you know what the consent round trip costs in wall-clock time.
The MCP Inspector is the fastest way to exercise both paths:
# From your server project root
npx @modelcontextprotocol/inspector node dist/server.js
The Inspector opens a local web UI with a connection panel and a tools panel. Two checks worth doing there:
- Approve path. Call
scrape_pagewithuse_js_render: trueagainsthttps://example.com/products. The Inspector renders the elicitation form. Approve. Confirm the tool result contains scrape data. - Decline path. Same call, decline the form. Confirm the tool result is the
fetch_not_authorizederror — and, more importantly, watch your server logs for the absence of an outbound request. If your logging shows the scrape API was contacted on a decline, your guard is in the wrong place.
For the latency question, time both configurations. Here are representative numbers from a local run against example.com, 20 iterations each, human approval clicks excluded (the operator clicked immediately):
| Configuration | Median total (ms) | p95 total (ms) | Elicitation overhead, median (ms) |
|---|---|---|---|
| Ungated fetch, plain GET | 1,840 | 3,120 | 0 |
| Gated fetch, plain GET | 2,010 | 3,590 | ~170 |
| Gated fetch, JS render | 6,480 | 9,900 | ~180 |
The elicitation round trip itself — server to client to server over local stdio or a local HTTP transport — adds roughly 150 to 200 ms. That is noise. The real cost is human reaction time, which will be anywhere from two seconds to never, and that’s exactly why the timeout from the previous section exists. Measure your own numbers with your transport; a remote client over a slow link will behave differently, and testing your MCP server before an agent depends on it pays for itself the first time a gate silently fails open.
One subtlety: exclude the human click from your overhead measurement, but include it in your operational expectations. They’re different numbers for different questions — “is my gate cheap?” versus “how long does a gated workflow take?”
Guardrails in Production: Rate-Limiting Elicitations and Avoiding Consent Fatigue
The failure mode of consent systems is not that people decline. It’s that people stop reading. The third dialog of the morning gets muscle-memory approval, and your expensive-fetch gate becomes a rubber stamp with extra latency. Design against that from day one.
The single most effective guardrail: cache approvals per domain. One confirmation per domain per session, not per URL.
// lib/approval-cache.ts
const approvedDomains = new Map<string, { until: number; maxCost: number }>();
const TTL_MS = 30 * 60 * 1000; // one session-ish
export async function gateFetch(
server: Server,
url: string,
cost: number
): Promise<boolean> {
const domain = new URL(url).hostname;
const cached = approvedDomains.get(domain);
if (cached && Date.now() < cached.until && cost <= cached.maxCost) {
return true; // prior informed consent covers this fetch
}
const result = await elicitWithTimeout(server, {
message: `Approve fetches to ${domain} (this one: ~${cost} tokens)?`,
requestedSchema: {
type: "object",
properties: {
approve: { type: "boolean", title: "Approve" },
remember: {
type: "boolean",
title: "Auto-approve this domain for 30 minutes?",
},
},
required: ["approve"],
},
});
if (result.action !== "accept" || result.content?.approve !== true) {
return false;
}
if (result.content.remember) {
approvedDomains.set(domain, { until: Date.now() + TTL_MS, maxCost: cost });
}
return true;
}
Two deliberate choices in that cache. The maxCost cap means a blanket approval for cheap fetches doesn’t silently cover a captcha-solving headful render later — cost escalation re-triggers the gate. And the TTL keeps approvals scoped to a session rather than a persistent config file, which preserves the whole point of asking in the first place.
The second guardrail is capability detection. Not every MCP client supports elicitation. If you call server.elicit() against a client that never declared the capability, behavior is undefined at best and a hang at worst. Check first:
function supportsElicitation(clientCapabilities: ClientCapabilities): boolean {
return Boolean(clientCapabilities?.elicitation);
}
// At tool-call time:
if (isCostly(input)) {
if (supportsElicitation(clientCapabilities)) {
const ok = await gateFetch(server, input.url, cost);
if (!ok) return consentError(input.url, "consent_declined");
} else {
// Fallback policy — pick one and be consistent
return {
isError: true,
content: [{
type: "text",
text: JSON.stringify({
error: "fetch_not_authorized",
reason: "elicitation_unsupported",
hint: "This client cannot confirm costly fetches. " +
"Run with an elicitation-capable client or reduce options.",
}),
}],
};
}
}
My recommendation for the fallback: fail closed for anything above your cost threshold. The alternative — letting costly fetches through because the client can’t ask — turns your entire consent system into an optional feature that silently disappears on half your clients. Fail closed is annoying. It’s also correct. If your agents mostly run unattended through clients without elicitation support, then your real answer is teaching the agent to escalate its own fetches conversationally, and elicitation becomes the attended-mode optimization rather than the only gate.
Wrap-Up
Elicitation turns scraping tools from “the agent fetched, here’s the bill” into “the agent asked, you decided.” The implementation is three pieces: a consent schema the client renders as a form, a handler where the fetch call physically sits after the approval check, and a timeout that turns silence into a structured error the model can act on instead of retrying blindly.
What to remember when you build it:
- Gate on estimated cost, not URL count, and cap cached approvals by cost so blanket yeses don’t cover premium fetches.
- Credentials elicited mid-run shrink your blast radius but don’t make secrets magically safe — they transit memory and the client.
- Distinguish
consent_declinedfromconsent_timeoutin errors and telemetry; one is a policy problem, the other is an availability problem. - Fail closed when the client lacks elicitation support, and measure the decline path locally with the Inspector before an agent ever depends on it.
The per-request overhead is trivial — a couple hundred milliseconds locally. The human in the loop is the expensive part, so spend their attention where it’s worth something: premium fetches, authenticated sessions, and domains that deserve a second look. Everything else should run straight through.
Related Articles
Teach Your Agent to Escalate Its Own Fetches
Wire MCP fetch tools in tiers so your agent tries a cheap plain fetch first, then escalates to JS rendering and geo-routing only when a page needs it.
TechnicalSlow Scrape, No Timeout: MCP Progress and Cancellation
A JS-rendered page can outlast your agent's tool call timeout. Add MCP progress notifications, handle cancellation, and stop paying for abandoned scrapes.
TechnicalThrottle MCP Tool Calls: A Token Bucket for Agents
AI agents can fire MCP scraping tools in bursts that burn quota and trip rate limits. Build a token-bucket throttle that paces tool calls without stalls.