Your Data Sits in an Iframe: Fix Extraction Across Frames
Your selector matches nothing, yet the data is visible in the browser. Here's why iframes break extraction and how to fetch frame content reliably.
On this page
The Price Is on the Screen — and Your Scraper Still Can’t See It
The price is on the screen. You can see it. Your selector cannot.
You copied .price straight out of DevTools, pasted it into your scraper, and got null. No exception, no warning — just an empty result where a dollar amount should be. Then you fetched the page with plain HTTP and grepped the response body: the string isn’t there either. The element renders in a browser, but neither your selector nor your raw HTML pull has ever seen it.
That combination — visible in the browser, absent from your DOM, absent from your fetched HTML — is the signature of an iframe. The data lives in a separate document, and nearly every extraction bug that follows comes from treating that document like part of the parent page. It isn’t. Here is why frames break extraction, and the three reliable ways to get at what’s inside them.
Why document.querySelector() Returns Null When the Data Is Clearly on Screen
An iframe is a browsing context, not a container element. When example.com/product/123 embeds a catalog widget from store.example.com, the parent DOM contains an <iframe> element node and nothing else from the frame. The frame’s entire markup lives in a second document, reachable only through iframe.contentDocument or contentWindow.document.
Selector engines search the document you hand them. In a page script, document.querySelector means window.document — the top-level one. .price could be the first child of the frame’s body and the parent’s querySelector would still return null, because the parent tree does not contain that node. Not hidden. Not lazy-loaded. Simply in a different tree.
| Call | Document searched | Result |
|---|---|---|
document.querySelector('.price') | Top-level document (example.com) | null — no .price node exists in this tree |
frame.contentDocument.querySelector('.price') | Frame document (store.example.com) | <div class="price">$42.99</div> |
// Console on https://example.com/product/123
document.querySelector('.price');
// → null
const frame = document.querySelector('iframe[src*="store.example.com"]');
if (frame.contentDocument) {
frame.contentDocument.querySelector('.price');
// → <div class="price">$42.99</div>
} else {
console.log('cross-origin: use fetch or automation');
}
Here’s the detail that costs people an afternoon: DevTools misleads you by being helpful. When you inspect an element inside a frame, the console’s execution context switches into that frame. So document.querySelector('.price') works in the console — you’re implicitly running it against the frame’s document. Your script runs in the top frame and fails. Same selector, different document. That mismatch is why “it works in the console but not in my code” is the most common iframe bug report, and why re-verifying the selector over and over never fixes anything.
The Same-Origin Gate: When Browsers Let You Read Frame Content and When They Block It
Before you write any extraction code, figure out which access case you’re in. Browsers decide frame DOM access by origin — scheme, host, and port. Host includes subdomain: store.example.com is a different origin from example.com, full stop.
| Embedding page | Frame origin | Same origin? | iframe.contentDocument | What you hit |
|---|---|---|---|---|
https://example.com | https://example.com/embed/list | Yes | Full document | Nothing — read away |
https://example.com | https://store.example.com | No (host differs) | null | SecurityError on contentWindow.document |
https://example.com | https://payments.other-site.com | No (host differs) | null | Same SecurityError |
http://example.com | https://example.com | No (scheme differs) | null | Same SecurityError |
const frame = document.querySelector('iframe[src*="payments.other-site.com"]');
frame.contentDocument;
// → null — cross-origin frames return null without throwing
try {
frame.contentWindow.document.body;
} catch (e) {
console.log(e.name);
// → "SecurityError"
console.log(e.message);
// → Blocked a frame with origin "https://example.com" from
// accessing a cross-origin frame.
}
Note the asymmetry. contentDocument fails silently with null; contentWindow.document throws. Code that branches on if (frame.contentDocument) handles both cases. Code that assumes contentWindow.document works everywhere will throw on the first cross-origin frame it meets.
Two consequences for scraping. First, if the frame is same-origin, you can read its DOM directly from the parent page — no browser automation needed. Second, if it’s cross-origin, no amount of parent-page JavaScript will ever see its DOM. Your options become: fetch the frame URL as its own request, or drive the frame through a browser automation client. Playwright and Puppeteer can read cross-origin frames because they talk to the browser over its debug protocol rather than running inside the parent page — the same-origin gate constrains page scripts, not the automation client.
(postMessage exists for cross-frame communication, but it requires the frame’s own code to cooperate and respond. For extraction, it’s a dead end.)
Finding Every Frame Before You Select From It: Enumerating iframes and frame Busting Trees
Guessing frame names or src patterns is how you end up with selectors that work on one page and fail on its sibling. Enumerate first. Select second.
function listFrames(doc = document, depth = 0, out = []) {
for (const frame of doc.querySelectorAll('iframe')) {
let origin = '(cross-origin, unreadable)';
try {
origin = new URL(frame.contentWindow.location.href).origin;
} catch {
// Thrown for cross-origin frames; fall back to parsing src
origin =
new URL(frame.getAttribute('src') ?? 'about:blank', location.href).origin +
' (unreadable)';
}
out.push({ depth, src: frame.getAttribute('src'), origin });
if (frame.contentDocument) {
// Recursion halts at cross-origin boundaries: contentDocument is null
listFrames(frame.contentDocument, depth + 1, out);
}
}
return out;
}
console.table(listFrames());
Run on https://example.com/product/123:
depth src origin
0 https://store.example.com/widget/list https://store.example.com (unreadable)
One frame at depth 0. But suppose the widget is known to nest a review panel inside itself — it doesn’t appear, and it can’t. The parent can read the store.example.com frame’s src attribute, but not its document, so recursion stops at the origin boundary. To map the next level, open https://store.example.com/widget/list as a top-level page and run the same helper there:
depth src origin
0 https://widget.example.net/panel https://widget.example.net (unreadable)
Full chain: example.com → store.example.com → widget.example.net. Two runs, because any single document can only show you the frames it can read, plus one level of src attributes.
One note on framebusting, since the heading raises it: some pages refuse to be embedded at all (if (top !== self) top.location = self.location). That matters if you try to load a data page inside your own framing harness. It doesn’t matter for scraping, where the target page is the top page and you attach to its frames from outside.
Fetching Frame Content Directly: Skipping the DOM with the Frame’s URL
Now the opinionated part. If the frame URL is stable and public, I don’t want frame-context switching at all — I want the frame’s URL fetched as its own request. The parent page is a shell. The data document is the src. Loading the shell to reach the document wastes a full page load, a JS bundle download, and a render, every single run.
The following snippet runs on Node 18 or later, where the global fetch is built in — no node-fetch install required:
const FRAME_URL = 'https://store.example.com/widget/list?page=2';
const res = await fetch(FRAME_URL, {
headers: {
'user-agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36',
// 'referer': 'https://example.com/',
// ^ Node/undici only — browsers reject this as a forbidden header.
// Uncomment if the frame checks the referring page.
'accept': 'text/html',
},
});
const html = await res.text();
// Regex is fine for flat, repeated tags. Anything nested: use a parser.
const titles = [...html.matchAll(/<h3 class="item-title">([^<]+)<\/h3>/g)].map(
(m) => m[1].trim()
);
console.log(titles.slice(0, 3));
// → [ 'Mechanical Keyboard', 'USB-C Hub', '27" Monitor' ]
Check the frame’s network traffic before you commit to this. Half the time the frame isn’t loading HTML at all — it’s calling a JSON endpoint, which is even better. Hidden JSON endpoints vs DOM parsing covers that hunt.
The same request through FineData’s /api/v1/async/scrape endpoint, for when the frame checks TLS fingerprints or blocks datacenter IPs. It’s an async job: the submit returns a job_id, which you poll via GET /api/v1/async/jobs/{job_id} (or receive at a callback_url):
curl -X POST https://api.finedata.ai/api/v1/async/scrape \
-H "Authorization: Bearer fd_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"url": "https://store.example.com/widget/list",
"formats": ["rawHtml"],
"extract_rules": {
"title": "h3.item-title",
"price": ".price"
},
"headers": { "referer": "https://example.com/" }
}'
Latency, from a local harness against test pages: the direct fetch of the frame URL consistently finishes in a fraction of the time a full page load takes, because it skips the shell page’s JavaScript bundle, layout, and render entirely. One request against one document beats one request plus a bundle download plus a render, every single run — and the gap only widens on real production pages with heavier scripts.
Closing: Match the Method to the Frame
The workflow, in order. First, confirm the data actually lives in an iframe — visible in the browser, absent from your DOM, absent from your fetched HTML is the signature. Second, check the origin gate: same-origin frames yield to contentDocument, cross-origin frames never will. Third, enumerate the frame tree before writing a single selector, so you know exactly which document holds the data. Fourth, if the frame URL is stable and public, skip the DOM entirely and fetch it as its own request — or hand it to FineData when fingerprinting or IP blocks get in the way.
Do those four steps in order, and the price on the screen stops being invisible: it becomes just another response body with a selector attached.
Related Articles
Your Prices Parse as NaN: Fix Number Extraction
Scraped prices like '$1,299.00' become NaN after type conversion. Trace the parsing failure, normalize messy number formats, and fix it before load.
TutorialYour Data Sits in a Shadow Root: Pierce It for Extraction
Selectors work in DevTools but return null in your scraper? Learn why web components hide data inside shadow roots and how to pierce them.
TutorialGzip Says Unexpected EOF: Fix Truncated Page Bodies
Your scraper dies mid-decompression with 'unexpected end of file'. Trace the truncated gzip stream back to its cause and learn fixes that hold up at scale.