Wikipedia Scraping Returns None: Fix the Page and the Selector

Why a Wikipedia Parser Returns None

A Wikipedia scraper returning None can have two different problems: it did not retrieve the article, or its CSS selector does not match the article’s current structure. Calling .get_text() on that missing match produces the familiar AttributeError: 'NoneType' object has no attribute 'get_text'. Adding a null check prevents the exception, but does not recover the missing content.

Consider a research tool that collects article titles and opening paragraphs for a searchable reading list. It needs a small, validated record, not an entire navigation menu or a silently empty description. Diagnose retrieval first, then selection, then text quality. We observed all three concerns while testing the public Web scraping article: a failed retrieval with a reported status of 200, a successful retrieval with a null lead field, and an extracted paragraph with joined words.

What the Live Requests Actually Showed

The checks below were performed on September 5, 2026 through FineData’s MCP interface. They are individual observations, not a benchmark or a claim about every Wikipedia page.

The initial request used use_antibot: false, use_js_render: false, auto_retry: false, max_retries: 1, timeout: 45, formats: ["markdown"], and only_main_content: true. The tool reported this error:

Request failed with status 200 (block_reason: stub_200_known_size)

Changing only use_antibot to true produced the same error. A subsequent request with use_antibot: false, use_js_render: true, and timeout: 60 succeeded: reported status 200, 6 tokens, and 4,358 ms. These are the tool’s reported values; we did not capture the outer HTTP response or a complete JSON error body during those checks.

The result does not establish that Wikipedia requires JavaScript. Switching retrieval modes also changes the request path; this evidence cannot identify why the first path returned a stub. Inspect your own error and content before changing modes. A browser request costs more than a plain request and is not a general answer to missing selectors, authentication failures, or rate limits.

The successful rendered response still included navigation despite only_main_content: true. That option is useful for reducing boilerplate, but the observation shows why a downstream index should validate specific fields rather than assume every Markdown response is a clean article.

A Successful Request Can Still Have a Missing Field

After rendering succeeded, the extraction rules below returned title: "Web scraping" and lead: null:

{
  "title": "#firstHeading",
  "lead": "#mw-content-text .mw-parser-output > p:not(.mw-empty-elt)"
}

The > combinator means a direct child. The rule looks for a paragraph immediately below .mw-parser-output. It cannot find a paragraph nested inside a section wrapper.

Parsoid’s HTML specification describes section wrappers. In this case, selecting through the lead section returned the missing paragraph:

{
  "title": "#firstHeading",
  "lead": "#mw-content-text .mw-parser-output section[data-mw-section-id=\"0\"] > p:not(.mw-empty-elt)"
}

This is a scoped fix, not a promise that one selector fits every page, skin, or parser version. Redirects, disambiguation pages, and pages without a conventional opening paragraph need explicit handling. A string extraction rule returns the first matching element; it does not collect all paragraphs. Store the source URL alongside the record so a failed assertion can be investigated later.

A Complete Python Request with Field Validation

Install httpx, export FINEDATA_API_KEY, and save the following as wikipedia_lead.py. It uses the successful retrieval settings and corrected selectors from the live check. The wrapper and validation logic are checked with synthetic responses in the repository; the live measurements above came from MCP, not this Python process.

There are three checks: outer API HTTP status, the JSON scrape result, and the extracted fields. An HTTP success alone is insufficient. The documented response includes a target status_code, but an error also needs its success flag and diagnostic metadata read together. Do not treat every failure as a target rate limit.

import os
import httpx

ARTICLE_URL = "https://en.wikipedia.org/wiki/Web_scraping"
TITLE_SELECTOR = "#firstHeading"
LEAD_SELECTOR = (
    '#mw-content-text .mw-parser-output '
    'section[data-mw-section-id="0"] > p:not(.mw-empty-elt)'
)


def validate_extraction(title, lead, min_lead_chars=40):
    if not isinstance(title, str) or not title.strip():
        raise ValueError("Missing article title; inspect the retrieved page")
    if not isinstance(lead, str) or len(lead.strip()) < min_lead_chars:
        raise ValueError("Missing or short lead; inspect the section selector")


def fetch_lead(api_key, *, client):
    response = client.post(
        "https://api.finedata.ai/api/v1/scrape",
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "url": ARTICLE_URL,
            "formats": ["markdown"],
            "only_main_content": True,
            "use_antibot": False,
            "use_js_render": True,
            "auto_retry": False,
            "max_retries": 1,
            "timeout": 60,
            "extract_rules": {
                "title": TITLE_SELECTOR,
                "lead": LEAD_SELECTOR,
            },
        },
    )
    # Check the API status before interpreting a target response envelope.
    if not 200 <= response.status_code < 300:
        raise RuntimeError(f"Scrape API HTTP {response.status_code}; stop")
    try:
        result = response.json()
    except ValueError as exc:
        raise RuntimeError("Scrape API returned non-JSON content") from exc
    if not isinstance(result, dict):
        raise RuntimeError("Scrape API returned an invalid envelope")
    status = result.get("status_code")
    if result.get("success") is not True:
        meta = result.get("meta")
        reason = meta.get("block_reason") if isinstance(meta, dict) else None
        raise RuntimeError(f"Scrape failed: reported status={status}, reason={reason}")
    if type(status) is not int or not 200 <= status < 300:
        raise RuntimeError(f"Unexpected reported status: {status}")
    data = result.get("data")
    fields = data.get("extract") if isinstance(data, dict) else None
    if not isinstance(fields, dict):
        raise ValueError("Response has no extracted fields")
    title, lead = fields.get("title"), fields.get("lead")
    validate_extraction(title, lead)
    return {"source_url": ARTICLE_URL, "title": title.strip(), "lead": lead.strip()}


if __name__ == "__main__":
    with httpx.Client(timeout=75) as client:
        print(fetch_lead(os.environ["FINEDATA_API_KEY"], client=client))

A title and a minimum paragraph length catch null and obviously short results; they do not prove semantic correctness. For a production reading list, compare the title with the requested page, retain a retrieval timestamp, and quarantine unexpected page shapes. These examples disable automatic retries. If a request is rate-limited, respect the service’s instructions rather than escalating through rendering modes. See the Steam 429 example for application-level handling of Retry-After.

Matching Text Is Not the Same as Clean Text

The corrected live extraction returned "Web scraping" as the title. Its lead began with this exact excerpt:

Web scraping,web harvesting, orweb data extractionisdata scrapingused forextracting datafromwebsites.[1]

The joined words matter for search and embeddings. Text extraction without separators between adjacent nodes can produce this pattern. We observed the joined output; we did not independently capture the precise source-node boundaries responsible for every join.

Once those boundaries have been discarded, a broader text selector cannot reconstruct them. For local normalization, request formats: ["rawHtml"] and use the HTML in data.raw_html. That request format is documented; a separate live raw-HTML normalization request was not part of these measurements. Parse the selected element, then join its text nodes with spaces:

from bs4 import BeautifulSoup


def normalized_lead(raw_html):
    soup = BeautifulSoup(raw_html, "html.parser")
    element = soup.select_one(LEAD_SELECTOR)
    if element is None:
        raise ValueError("Lead paragraph is absent from this HTML")
    return element.get_text(" ", strip=True)

This helper belongs below the first example’s selector constants and requires beautifulsoup4. It operates on HTML, not the already joined lead string. Adding separators can also add spaces around punctuation and reference markers, so inspect the output before adopting further normalization. The API result above should not be presented as if it already contained this client-side cleanup.

Test the Structural Failure without More Requests

Save the helpers and this synthetic fixture in a separate local test file, without the network-calling if __name__ == "__main__" block. It checks the selector and spacing mechanism without making a request. The fixture deliberately isolates the section wrapper and inline text nodes; it is not captured Wikipedia HTML and makes no claim about live availability.

DIRECT_HTML = """
<div id="mw-content-text"><div class="mw-parser-output">
<p>Web scraping is a method of collecting data from public web pages.</p>
</div></div>
"""
SECTION_HTML = """
<div id="mw-content-text"><div class="mw-parser-output">
<section data-mw-section-id="0"><p><b>Web scraping</b><a>extracts data</a>
from web pages for a local research index.</p></section>
</div></div>
"""
OLD_SELECTOR = '#mw-content-text .mw-parser-output > p:not(.mw-empty-elt)'
assert BeautifulSoup(DIRECT_HTML, "html.parser").select_one(OLD_SELECTOR) is not None
assert BeautifulSoup(SECTION_HTML, "html.parser").select_one(OLD_SELECTOR) is None
assert normalized_lead(SECTION_HTML).startswith("Web scraping extracts data")

The regression suite also executes the article’s actual Python blocks against mocked API responses: outer HTTP failures, non-JSON content, a failed scrape reporting 200, missing fields, and valid extraction. No test needs a Wikipedia request. Keeping tests attached to the published snippets makes a later edit to a selector or response path visible instead of silently diverging from the tutorial.

Prefer Structured Sources for Larger Collections

FineData is useful when you need the rendered page and a particular DOM field. If you only need Wikipedia text or metadata, evaluate the Action API first. It avoids coupling your application to a page skin. For bulk datasets, consider Wikimedia dumps instead of repeated rendered requests.

Follow Wikimedia’s User-Agent policy and API etiquette when building a client. Review the Terms of Use, including text reuse: attribution and applicable share-alike requirements still matter whether you collect through HTML, an API, or a dump. An internal research label is not a blanket exemption from those obligations.

For this problem, the useful sequence is concrete: verify retrieval, match the actual section structure, reject missing fields, and normalize from HTML when node boundaries matter. Rendering solved one observed retrieval failure; it did not solve the selector or spacing problems for us.