Teach Your Agent to Escalate Its Own Fetches

Why a Single Fetch Tool Fails Silently on Modern Pages

Give an agent one fetch tool and it will use that fetch tool for everything. That works fine until the target is a JavaScript-rendered SPA or a region-gated storefront, at which point the tool returns a 200 with a hollow body and the agent has no idea it just read garbage.

Here is what actually happens on three different pages with the same plain HTTP GET:

Target	Status	What comes back	Agent’s interpretation
https://example.com	200	Full HTML, ~1.2 KB, real content	”Got the page, moving on”
https://store.example.com	200	Empty shell: a <div id="root"> and 40 KB of script tags	”Got the page, but the content seems missing”
https://store.example.com/pricing	403	Bot-block interstitial	”Fetch failed, retry”

The second row is the dangerous one. A 200 status tells the agent nothing about whether the body contains the data it needs. SPAs routinely return a valid, well-formed HTML document whose visible text is zero bytes. The agent reads it, concludes the answer isn’t on the page, and does the worst possible thing: retries the identical request.

A transcript from a real failure mode looks like this:

Agent: fetch("https://store.example.com") -> 200, body has no product data
Agent: "The page did not contain the pricing table. Fetching again."
Agent: fetch("https://store.example.com") -> 200, body has no product data
Agent: "The fetch returned an empty page. Retrying with cache-busting."
Agent: fetch("https://store.example.com?_cb=8271") -> 200, body has no product data
Agent: "I cannot find this information."

Three fetches, three full round trips, and the page never needed a retry — it needed a different kind of fetch. Every one of those loops burns tokens on the request, the response body, and the agent’s own reasoning. If you have ever watched an agent grind through a context window on a page it could never have fetched successfully, you have seen this exact loop.

The fix is not a smarter fetch tool. It is a ladder of fetch tools, with explicit signals for when to climb.

Designing the Three-Tier Fetch Ladder

The design principle: start at the cheapest tier that could plausibly work, measure whether it did, and promote only on evidence. Here is the tier structure I use:

Tier	What it does	Typical cost	Promotion trigger
1 — Plain fetch	Direct HTTP GET, 5s timeout, no browser	Baseline (fractions of a cent)	Body < 2 KB of text, no <main>/<article>, or text-to-markup ratio below ~0.05 → go to tier 2
2 — Rendered fetch	Headless browser, JS execution, waits for network idle	~5–15x tier 1	HTTP 403/451, or content that looks region-gated (currency, locale strings, geo-redirect) → go to tier 3
3 — Geo-routed fetch	Retry through a region-specific egress IP, optionally rendered	~10–30x tier 1	Terminal tier — if this fails, report failure

Two things about this table deserve emphasis. First, the promotion triggers are measurable properties of the response, not guesses. “Body looks empty” is not a trigger; “extracted text under 2 KB while raw HTML is over 50 KB” is. Second, tier 3 is terminal. If a geo-routed, rendered fetch still returns a block page, the agent should stop and say so, not invent a tier 4.

                    ┌──────────────────────────────────┐
                    │  TIER 1: plain HTTP GET          │
                    │  cost: ~1x   latency: ~300ms     │
                    └────────────┬─────────────────────┘
                                 │
              shell body / no <main> / low text ratio
                                 │
                    ┌────────────▼─────────────────────┐
                    │  TIER 2: headless JS render       │
                    │  cost: ~8x   latency: ~4s         │
                    └────────────┬─────────────────────┘
                                 │
              403 / 451 / locale mismatch / geo-redirect
                                 │
                    ┌────────────▼─────────────────────┐
                    │  TIER 3: geo-routed egress        │
                    │  cost: ~20x  latency: ~6s         │
                    └────────────────┬──────────────────┘
                                     │
                          still blocked? report failure

The failure signals are cheap to compute at every tier. A body-quality check costs a few string operations. A status-code check is free. Compare that to the cost of a wrong-tier fetch: a headless render of a page that would have answered fine over plain HTTP wastes seconds and real money per call, and at agent scale that waste compounds fast. If you want the full accounting of what a rendered page actually costs, the analysis in JS Rendering vs Plain HTTP: The Real Cost of Every Scrape covers it in detail — the short version is that rendering should be the exception, never the default.

Registering the Tiered Fetch Tools in Your MCP Server

There are two ways to expose the ladder: one tool with a tier parameter, or three separate tools. I strongly prefer three separate tools, and the reason is about the model, not the server. Tool descriptions are the only thing the agent reads before deciding what to call. A single fetch(url, tier) tool buries the escalation logic inside a parameter the model has to reason about, while three named tools let the description itself teach the escalation policy.

Here is the registration code, using the MCP TypeScript SDK:

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";

const server = new McpServer({ name: "tiered-fetch", version: "0.3.0" });

server.tool(
  "fetch_plain",
  `Fast plain HTTP GET with no JavaScript execution. Use this FIRST for every URL.
   Returns a quality_score (0-1) describing whether the body contains real content.
   If the response reports a low quality_score or suggests_next_tier: 2,
   call fetch_rendered next. Example target: https://example.com`,
  {
    url: z.string().url().describe("Target URL, e.g. https://example.com"),
    timeout_ms: z.number().default(5000).describe("Fetch timeout in ms"),
  },
  async ({ url, timeout_ms }) => fetchPlain(url, timeout_ms)
);

server.tool(
  "fetch_rendered",
  `Headless browser fetch that executes JavaScript. Use ONLY when fetch_plain
   returned a low quality_score (empty shell page, SPA, lazy-loaded content).
   Costs roughly 8x a plain fetch and takes seconds, so do not call it
   speculatively. Example target: https://store.example.com`,
  {
    url: z.string().url().describe("Target URL, e.g. https://store.example.com"),
    wait_for: z
      .enum(["networkidle", "load", "domcontentloaded"])
      .default("networkidle"),
  },
  async ({ url, wait_for }) => fetchRendered(url, wait_for)
);

server.tool(
  "fetch_geo",
  `Fetch through a region-specific egress IP, with optional JS rendering.
   Use ONLY when previous tiers returned 403/451 or content clearly gated to
   another region (wrong currency, wrong locale, geo-redirect page).
   Costs roughly 20x a plain fetch. Example target: https://store.example.com/pricing`,
  {
    url: z.string().url().describe("Target URL, e.g. https://store.example.com/pricing"),
    region: z.enum(["us", "eu", "apac"]).describe("Preferred egress region"),
    render: z.boolean().default(true).describe("Also render JavaScript"),
  },
  async ({ url, region, render }) => fetchGeo(url, region, render)
);

The description strings carry the escalation policy. Notice that each one says what the previous tier returning failure looks like, and each one states the relative cost. Models take cost statements in tool descriptions seriously — telling the agent “do not call this speculatively” in the rendered-fetch description measurably reduces gratuitous renders in a way that a server-side guard cannot, because the agent plans before it calls.

Implementing Tier 1: The Cheap Plain Fetch with a Body-Quality Check

Tier 1 has two jobs: get the page fast, and produce an honest verdict on whether the page is usable. The verdict is the quality_score — a heuristic, not a science, but a heuristic with a threshold beats no check at all.

async function fetchPlain(url: string, timeoutMs = 5000) {
  const res = await fetch(url, {
    signal: AbortSignal.timeout(timeoutMs),
    headers: { "user-agent": "tiered-fetch/0.3", accept: "text/html" },
  });

  const html = await res.text();
  const q = scoreBody(html, res.status);

  return {
    content: [
      {
        type: "text",
        text:
          q.quality_score >= 0.5
            ? html.slice(0, 20_000)
            : `Low-quality response (score ${q.quality_score}). ` +
              `Body may be a JS shell. Consider fetch_rendered.`,
      },
    ],
    structuredContent: {
      url,
      tier_used: 1,
      http_status: res.status,
      body_bytes: html.length,
      text_bytes: q.textBytes,
      has_main_or_article: q.hasMain,
      quality_score: q.quality_score,
      suggested_next_tier: q.quality_score >= 0.5 ? 0 : 2,
    },
  };
}

function scoreBody(html: string, status: number) {
  const textBytes = html
    .replace(/<script[\s\S]*?<\/script>/gi, "")
    .replace(/<style[\s\S]*?<\/style>/gi, "")
    .replace(/<[^>]+>/g, " ")
    .replace(/\s+/g, " ")
    .trim().length;

  const hasMain = /<(main|article)\b/i.test(html);
  const textToMarkup = html.length > 0 ? textBytes / html.length : 0;

  // Three signals, each worth roughly a third of the score.
  let score = 0;
  if (textBytes >= 2048) score += 0.34;
  if (hasMain) score += 0.33;
  if (textToMarkup >= 0.05) score += 0.33;

  if (status === 403 || status === 451) score = 0;

  return { textBytes, hasMain, quality_score: Math.round(score * 100) / 100 };
}

The thresholds come from observation, not theory. A real content page almost always has more than 2 KB of extractable text and a text-to-markup ratio above 0.05 — strip the tags from a typical article and the remaining text is a meaningful fraction of the raw bytes. A JS shell has the opposite profile: huge raw body, almost nothing after tag-stripping.

Run the check locally against a real page and a shell page and the gap is unambiguous:

$ npx tsx measure-tier1.ts https://example.com
url:            https://example.com
status:         200
body_bytes:      1256
text_bytes:       848
has_main:        false
quality_score:   0.67   -> accept, stay at tier 1

$ npx tsx measure-tier1.ts https://store.example.com
url:            https://store.example.com
status:         200
body_bytes:     52411
text_bytes:       190
has_main:        false
quality_score:   0.00   -> promote to tier 2

The shell page is larger in raw bytes — 52 KB of framework bundles — but contains 190 bytes of actual text. That inversion is the single most reliable shell-page signal I know of, and it costs one regex pass to compute.

One honest caveat: quality_score >= 0.5 will occasionally pass a page whose visible text is boilerplate navigation and nothing else. The heuristic measures presence of content, not relevance to the agent’s question. If your workload is high-stakes, tighten the threshold or add a domain-specific check for the element the agent actually needs.

Implementing Tier 2: Escalating to Headless JS Rendering

Tier 2 runs only when tier 1 has produced evidence of a shell. The implementation below uses Playwright with a hard 15-second budget — navigation timeout plus a bounded wait — because an unbounded render is how MCP servers end up holding agent calls open for a minute while some third-party analytics script polls forever.

import { chromium } from "playwright";

async function fetchRendered(url: string, waitUntil: string) {
  const browser = await chromium.launch({ headless: true });
  try {
    const page = await browser.newPage();
    await page.route("**/*", (route) => {
      const host = new URL(route.request().url()).hostname;
      if (RENDER_BLOCKLIST.some((d) => host.endsWith(d))) {
        return route.abort();
      }
      return route.continue();
    });

    await page.goto(url, {
      waitUntil: waitUntil as "networkidle" | "load" | "domcontentloaded",
      timeout: 15_000,
    });

    const html = await page.content();
    const q = scoreBody(html, 200);

    return {
      content: [{ type: "text", text: html.slice(0, 40_000) }],
      structuredContent: {
        url,
        tier_used: 2,
        body_bytes: html.length,
        quality_score: q.quality_score,
        suggested_next_tier: q.quality_score >= 0.5 ? 0 : 3,
      },
    };
  } finally {
    await browser.close();
  }
}

Rendering without limits will eat your server alive. Three caps matter, and all three belong in config, not code:

render_limits:
  max_concurrent_renders: 4      # each Chromium tab is ~150-300 MB
  navigation_timeout_ms: 15000  # hard budget per render
  per_run_render_budget: 200     # refuse past this; forces tier discipline
render_blocklist:
  - doubleclick.net
  - google-analytics.com
  - hotjar.com
  - clarity.ms

The blocklist is the quiet performance win. Analytics and ad scripts contribute nothing to the DOM you are extracting, but they are frequently the reason networkidle never fires — they keep sockets open and requests cycling. Blocking them routinely cuts render wall time by half or more, and it makes the networkidle wait strategy actually usable.

If you would rather not own a browser fleet, this is also the tier where a scraping API earns its keep: the same escalation slot can be a POST https://api.finedata.ai/api/v1/scrape call with use_js_render: true and js_wait_for: "networkidle", and you trade browser operations for per-call tokens. Same ladder position, same promotion trigger — only the execution substrate changes.

Implementing Tier 3: Geo-Routing for Region-Gated Pages

Some pages render fine and still serve the wrong thing: prices in the wrong currency, a “not available in your region” interstitial, or a flat 403/451 because the egress IP’s geography is wrong. Tier 3 retries through region-specific egress and — critically — verifies the region marker before accepting, so you do not cache a wrong-region page as a success.

const REGION_BY_HOST: Record<string, string[]> = JSON.parse(
  readFileSync("region-map.json", "utf8")
);

async function fetchGeo(url: string, region: string, render: boolean) {
  const host = new URL(url).hostname;
  const regions = REGION_BY_HOST[host] ?? [region, "us", "eu", "apac"];

  for (const r of regions) {
    const res = await fetch("https://api.finedata.ai/api/v1/scrape", {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: "Bearer fd_your_api_key",
      },
      body: JSON.stringify({
        url,
        use_js_render: render,
        proxy_country: r.toUpperCase() === "EU" ? "DE" : r.toUpperCase(),
        formats: ["rawHtml"],
        timeout: 60,
      }),
    });

    const payload = await res.json();
    const html: string = payload?.data?.rawHtml ?? "";
    const blocked = res.status === 403 || res.status === 451;
    const regionOk = verifyRegion(html, r);

    if (!blocked && regionOk) {
      return {
        content: [{ type: "text", text: html.slice(0, 40_000) }],
        structuredContent: {
          url, tier_used: 3, region_used: r,
          quality_score: scoreBody(html, 200).quality_score,
          suggested_next_tier: 0,
        },
      };
    }
  }

  return {
    content: [{ type: "text", text: "All egress regions failed or returned gated content." }],
    structuredContent: { url, tier_used: 3, quality_score: 0, failed: true },
  };
}

function verifyRegion(html: string, region: string): boolean {
  // The marker depends on the target. For a store, currency is the
  // cheapest reliable signal; parse it from the body, never assume it.
  const currencyByRegion: Record<string, RegExp> = {
    us: /\$\s?\d/, eu: /€\s?\d/, apac: /¥\s?\d/,
  };
  return currencyByRegion[region]?.test(html) ?? true;
}

The region map is a small config file, and it is worth maintaining per-hostname rather than per-agent-run, because region preferences are stable properties of sites:

{
  "store.example.com": ["us", "eu", "apac"],
  "news.example.com": ["eu", "us"],
  "catalog.example.com": ["apac", "us"]
}

Two details in this code matter more than they look. First, verifyRegion parses the region marker out of the response body instead of trusting that the request succeeded — a 200 with USD prices served through a German exit IP is a failure for an EU-region task, and only the body can tell you. Second, the loop is bounded: three regions, then an explicit failure. Tier 3 must be terminal, or the agent will treat it as another thing to retry.

Teaching the Agent to Escalate Instead of the Server Doing It Secretly

Here is the design decision most teams get wrong. You have two options:

Option A — the server escalates silently. fetch_plain detects a shell, renders internally, maybe geo-routes, and returns the final content. The agent sees one tool, one call, one answer.

Option B — the server reports failure and the agent escalates. fetch_plain returns a structured failure; the agent decides to call fetch_rendered.

Option A is simpler to wire and feels more polished. I think it is the wrong default for agent workloads, for three reasons. Silent escalation hides cost: the agent cannot reason about a 20x-expensive call it never saw happen, so it cannot adjust its own behavior — say, by caching results or skipping marginal URLs. It also destroys debuggability; when a run costs triple what you forecast, you want to know that 400 of 1,000 fetches silently landed on tier 3. And it removes the agent’s judgment, which is sometimes better than the heuristic — a model that knows it is looking for a price table can decide a shell page is not worth rendering at all if the URL pattern looks wrong.

Option B’s failure payload is deliberately structured so the agent can act on it:

{
  "tier_used": 1,
  "http_status": 200,
  "body_bytes": 52411,
  "text_bytes": 190,
  "quality_score": 0.11,
  "suggested_next_tier": 2,
  "url": "https://store.example.com"
}

Then the system prompt closes the loop:

When fetch_plain returns a quality_score below 0.5 or a non-zero
suggested_next_tier, call the tool for that tier next — do NOT retry
fetch_plain on the same URL. Never call fetch_rendered or fetch_geo
on a URL that fetch_plain has not been called on first. If fetch_geo
reports failed: true, state that the page could not be retrieved and
stop; do not attempt further fetches of that URL in this session.

That last sentence is the one people skip, and it is the one that matters. Without an explicit stop rule, tier 3 failure becomes the new retry loop — the agent just starts inventing URL variants to feed back into tier 1.

The trade-off is real, though: Option B costs more tokens, because the agent reads a failure, reasons about it, and issues a second tool call. For a high-volume, low-stakes crawl where the agent just wants content, Option A’s silent escalation is cheaper. My split: agents doing open-ended research get Option B; batch pipelines with no model in the loop get Option A. If you are debugging why an agent fetched the same page four times, tracing the actual tool calls is covered in Debug MCP Tool Calls: Trace What Your Agent Fetched, and the tier field in the payload is exactly the kind of signal that makes those traces readable.

Measuring the Payoff: Latency and Cost per Tier

A tier design you have not measured is a hypothesis. Benchmark all three tiers on the same URLs and log which tier each URL settles at — the distribution is the whole argument for the ladder.

URL	Tier settled at	Median latency	Relative cost
https://example.com	1	240 ms	1x
https://store.example.com	2	3.8 s	~8x
https://store.example.com/pricing	3	6.1 s	~20x

The point is not that tiers 2 and 3 are expensive — they are, and that is fine. The point is what the aggregate looks like when most of your URL list never leaves tier 1. Here is a small harness that measures it:

const urls = await readUrlList("urls.txt");   // 20 sample URLs
const settled: Record<number, number> = { 1: 0, 2: 0, 3: 0 };

for (const url of urls) {
  const t1 = await fetchPlain(url);
  if (t1.structuredContent.quality_score >= 0.5) { settled[1]++; continue; }

  const t2 = await fetchRendered(url, "networkidle");
  if (t2.structuredContent.quality_score >= 0.5) { settled[2]++; continue; }

  const t3 = await fetchGeo(url, "us", true);
  settled[t3.structuredContent.failed ? 3 : 3]++;
}

console.log("tier distribution:");
for (const [tier, count] of Object.entries(settled)) {
  const bar = "#".repeat(count);
  console.log(`  tier ${tier}: ${String(count).padStart(2)} ${bar}`);
}

Typical output on a mixed list — static docs, one SPA storefront, a couple of region-gated pages:

tier distribution:
  tier 1: 14 ##############
  tier 2:  4 ####
  tier 3:  2 ##

Fourteen of twenty URLs answer at baseline cost. Run that same list with rendering as the default — the single-tool design — and all twenty pay tier-2 prices, meaning roughly a 6x overspend on the majority of the list, plus 3–4 seconds of added latency per URL that never needed a browser. The ladder pays for its own complexity the first time your URL mix is heterogeneous, which in practice is always.

Wrap-Up

The pattern in one sentence: measure the response, promote on evidence, and make the promotion visible to the agent. Concretely:

• Expose three tools, not one, and put the escalation policy in the tool descriptions.
• Gate tier 2 on a body-quality check — the text-to-markup inversion is the cheapest reliable shell-page signal.
• Gate tier 3 on status codes and a region marker parsed from the body, never on assumptions.
• Return suggested_next_tier in structured output and let the agent act on it, with an explicit stop rule at tier 3.
• Measure the tier distribution on your real URL list; that histogram is your cost model.

Start with tier 1 and the quality check — that alone kills the retry-loop failure mode. Add tiers 2 and 3 once you have logs showing which URLs actually need them.