Fix JSONDecodeError: When Your Fetch Returns HTML, Not JSON

When a 200 Response Isn’t JSON Anymore

Your pipeline ran fine for three weeks. Then at 2 a.m. it threw json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) on a URL that returned perfect JSON an hour earlier. Nothing changed in your code. The URL still responds. The status code — checked it — was 200. So what happened?

The answer is almost always the same: the response body wasn’t JSON anymore. It was an HTML error page from a reverse proxy, a login redirect, or a WAF challenge, and json.loads() choked on the first character. The status code lied to you, or you never checked it at all. This post walks through why this happens, how to detect it before the parser crashes, and how to build a fetch wrapper that fails with an error message you can actually act on.

Why JSONDecodeError at Line 1 Column 1 Almost Always Means HTML Came Back

The traceback looks like this:

Traceback (most recent call last):
  File "pipeline.py", line 42, in fetch_products
    data = json.loads(response.text)
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value)
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

“Line 1 column 1 (char 0)” is the misleading part. It doesn’t mean your JSON has a syntax error on the first line. It means the parser looked at character zero and found something that cannot begin any valid JSON document. Valid JSON starts with one of a few characters: { for objects, [ for arrays, " for strings, or a digit/-/t/f/n for numbers and literals. An HTML page starts with <. That single byte kills the parse before anything else happens.

Compare the two bodies your endpoint can hand you:

	Real JSON response	HTML 502 error page
First characters	[**{**"id": 123, "name": "Widget", ...	[**<**!DOCTYPE html>\n<html>\n<head><title>502 ...
First byte	{	<
Content-Type	application/json	text/html
Parses with json.loads()	Yes	No — dies at char 0
What your code sees	A product list	A proxy’s apology

The column-1 position is your friend, oddly enough. When the error points at char 0, the body isn’t malformed JSON with a stray comma on line 40 — the body is a different document type entirely. That distinction should route your debugging: malformed JSON is a parsing bug on your side; char-0 failure means the server (or something between you and it) sent something else.

One more detail worth knowing: json.loads("") produces the exact same error. An empty body also fails at char 0. So “line 1 column 1” covers two cases — a non-JSON body and no body at all. Your logging needs to distinguish between them, which is why capturing the body on failure matters so much later in this post.

The Usual Suspects: Reverse Proxies, 302 Redirects, and WAF Pages That Return HTML with Status 200

Knowing that HTML arrived is half the diagnosis. The other half is figuring out who sent it. There are four common sources, and each has a fingerprint.

Failure source	Typical status	Body signature
Reverse proxy error (nginx 502/504)	502, 504	<html><head><title>502 Bad Gateway</title>... — static, mentions nginx
302 redirect to login	302 → 200	Small HTML page with a <meta http-equiv="refresh"> or a login form; final URL is not your API path
WAF / anti-bot challenge	403, 429, or 200	<script> blocks with challenge tokens, “checking your browser” copy
SPA fallback route	200	Your frontend’s index.html — full HTML shell, contains your JS bundle tag, zero API data

The last two are the dangerous ones. A challenge page can arrive with a 200 status, which is why “I check response.status_code == 200” is not a defense — a point covered in detail in Your 200 OK Responses Are Actually Block Pages. And SPA fallback routes are sneaky in the opposite direction: the server is healthy, the route just doesn’t exist, and the frontend router catches /api/* and serves the app shell with a friendly 200.

You can reproduce the redirect case by hand:

curl -sIL https://store.example.com/api/products

# HTTP/1.1 302 Found
# Location: https://store.example.com/login?next=/api/products
# HTTP/1.1 200 OK
# Content-Type: text/html; charset=utf-8

curl -s https://store.example.com/api/products | head -c 120
# <!DOCTYPE html><html><head><title>Sign in — store.example.com</title>...

The API path redirected to a login page. requests follows redirects by default, so response.text contains the HTML login form and response.status_code is a cheerful 200. Your parser then dies with the column-1 error and the log tells you nothing about the four-hop journey the request took.

The diagnostic move, every time: log response.url (the final URL after redirects), not just the URL you requested. If you asked for /api/products and ended up at /login, the problem is session handling, not JSON. If the final URL matches but the body starts with <, you’re looking at a proxy error or a challenge.

Sniffing the Body Before Parsing: A One-Line Content-Type and First-Byte Check

The fix for the crash itself is embarrassingly cheap. Check the Content-Type header and the first non-whitespace byte before you hand the body to the parser.

Python:

import requests

resp = requests.get(
    "https://store.example.com/api/products",
    headers={"Accept": "application/json"},
    timeout=30,
)

content_type = resp.headers.get("Content-Type", "")
first_byte = resp.text.lstrip()[:1]

if first_byte == "<" or "html" in content_type.lower():
    raise RuntimeError(
        f"Expected JSON, got HTML: status={resp.status_code}, "
        f"type={content_type}, url={resp.url}"
    )

data = resp.json()

Two checks, not one, and here’s why. Content-Type alone fails because misconfigured servers send text/plain or omit the header on valid JSON responses — I’ve seen APIs that do this in production for years. The first byte alone fails because a valid JSON response could theoretically start with whitespace… which lstrip() handles. But there’s a subtler case: some error payloads are valid JSON like {"error": "unavailable"}, which passes the first-byte check and parses cleanly. You’ll deal with that in the schema validation section below. For pure “is this HTML,” the first-byte check is the one that never lies.

The JavaScript equivalent, since fetch happily parses nothing for you:

const resp = await fetch("https://store.example.com/api/products");
const text = await resp.text();

if (text.replace(/^\s+/, "")[0] === "<" || resp.headers.get("content-type")?.includes("html")) {
  throw new Error(`Expected JSON, got HTML: status=${resp.status}, url=${resp.url}`);
}

const data = JSON.parse(text);

Note the deliberate choice to call resp.text() and then JSON.parse(), rather than resp.json(). resp.json() swallows the body — once it throws, you can’t log what the server actually sent, because the body stream is consumed. Reading text first costs you nothing and preserves the evidence.

Logging the Body Safely: Truncation, Redaction, and Structured Error Context

Detecting the HTML is step one. Step two is logging it well, because the body is your only evidence of what upstream did. Two failure modes to avoid here: logging nothing (you’re blind) and logging the entire body (a challenge page can be 500 KB of JavaScript, and your log pipeline will not thank you).

Redaction matters too. If the offending body happens to be a login page echo, or if you dump response headers alongside it, you can leak Set-Cookie values and authorization headers into logs that other people read. I’ve seen this happen. Strip before you log.

import re
import json
import logging

logger = logging.getLogger("pipeline")

SENSITIVE = re.compile(
    r'(authorization|cookie|set-cookie|Authorization)\s*[:=]\s*\S+',
    re.IGNORECASE,
)

def log_bad_body(resp, label: str) -> None:
    snippet = resp.text[:500]
    snippet = SENSITIVE.sub(r"\1=REDACTED", snippet)
    truncated = len(resp.text) > 500
    logger.error(
        "non_json_response",
        extra={
            "event": "non_json_response",
            "label": label,
            "status_code": resp.status_code,
            "final_url": resp.url,
            "content_type": resp.headers.get("Content-Type", ""),
            "body_length": len(resp.text),
            "body_preview": snippet + ("…[truncated]" if truncated else ""),
        },
    )

And what a JSON-lines log entry looks like when a 502 strikes:

{"event": "non_json_response", "label": "store_products", "status_code": 502, "final_url": "https://store.example.com/api/products", "content_type": "text/html", "body_length": 153, "body_preview": "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"}

One log line, and you already know three things: nginx sent it (upstream app was down), the URL was the real API path (not a redirect), and the status was 502 (not a challenge). That’s a complete diagnosis from a single error line, versus a raw JSONDecodeError traceback that tells you nothing.

A note on 500 characters: that’s usually enough to identify the page type from the <title> tag and any server banner. If your targets use challenge pages with inline scripts before any title, bump it to 1000. Don’t go higher — identification is the goal, not reproduction.

Failing Fast with a Custom Exception: Wrapping Fetch Calls in a Safe JSON Fetcher

The sniff-and-check code doesn’t scale as inline logic. Every fetch site in your codebase needs the same defense, and copy-pasting five lines of guards into 30 functions is how the guards rot. Wrap it once:

import json
import requests

class NonJsonResponseError(Exception):
    """The server returned a body that is not JSON."""
    def __init__(self, message, status, url, headers, body_preview):
        super().__init__(message)
        self.status = status
        self.url = url
        self.headers = dict(headers)
        self.body_preview = body_preview

def safe_json_fetch(url: str, *, label: str = "", **kwargs) -> dict:
    kwargs.setdefault("headers", {}).setdefault("Accept", "application/json")
    kwargs.setdefault("timeout", 30)
    resp = requests.get(url, **kwargs)

    content_type = resp.headers.get("Content-Type", "")
    first_byte = resp.text.lstrip()[:1]

    if first_byte == "<" or "html" in content_type.lower():
        preview = resp.text[:500]
        raise NonJsonResponseError(
            f"[{label or url}] expected JSON, got HTML "
            f"(status={resp.status_code}, content_type={content_type}, "
            f"final_url={resp.url}, first_byte={first_byte!r})",
            status=resp.status_code,
            url=resp.url,
            headers=resp.headers,
            body_preview=preview,
        )

    if not resp.text.strip():
        raise NonJsonResponseError(
            f"[{label or url}] empty response body "
            f"(status={resp.status_code}, final_url={resp.url})",
            status=resp.status_code,
            url=resp.url,
            headers=resp.headers,
            body_preview="",
        )

    try:
        return resp.json()
    except json.JSONDecodeError as exc:
        raise NonJsonResponseError(
            f"[{label or url}] malformed JSON at "
            f"line {exc.lineno} column {exc.colno} "
            f"(status={resp.status_code}, final_url={resp.url})",
            status=resp.status_code,
            url=resp.url,
            headers=resp.headers,
            body_preview=resp.text[:500],
        ) from exc

Notice the distinction in the third branch. A JSONDecodeError at line 1 column 1 with a non-< first byte means the body was some other non-JSON text or empty — handled above. A decode error at line 12 column 40 is a genuinely malformed payload, possibly a truncated response. Different causes, different messages, same exception type so callers catch one thing.

Compare what you get when the 502 hits:

	Raw json.loads()	safe_json_fetch()
Message	Expecting value: line 1 column 1 (char 0)	[store_products] expected JSON, got HTML (status=502, content_type=text/html, final_url=https://store.example.com/api/products, first_byte='<')
Status code	Not available	Attached to exception
Body evidence	Not available	500-char preview attached
Root cause guessable from log	No	Yes, in one line

The wrapper also centralizes retry policy decisions later. A 502 from the proxy is retryable. A redirect to login is not — you need to fix session handling. A challenge page requires a different fetch strategy entirely, which is exactly the escalation logic covered in Your Scraper Died in a Redesign: An Extraction Escalation. You can’t route retries by cause if your fetch layer throws one undifferentiated exception.

Validating the Payload Shape After Parsing: Catching JSON That Is Technically Valid but Structurally Wrong

Here’s the case the byte-sniffing misses: the API’s own error handler returned valid JSON. {"error": "unavailable", "code": 503} parses without complaint, passes the first-byte check, and then your downstream code crashes with a KeyError: 'products' three functions later — an error whose stack trace points nowhere near the actual problem.

Shape validation closes that gap. Pydantic is the pragmatic choice in Python because you probably want typed models anyway:

from pydantic import BaseModel, ValidationError

class Product(BaseModel):
    id: int
    name: str
    price: float

class ProductList(BaseModel):
    products: list[Product]

try:
    payload = ProductList.model_validate(data)
except ValidationError as exc:
    # data parsed as JSON but is the wrong shape —
    # e.g. {"error": "unavailable"} when we expected {"products": [...]}
    raise RuntimeError(
        f"payload shape mismatch for {label}: {exc.errors()[:1]} "
        f"— body starts with: {json.dumps(data)[:200]}"
    ) from exc

The full decision tree, condensed:

Condition	Verdict	Action
Body first byte is <	HTML error page	Log body, check status/final URL, fix session or escalate fetch
Body parses but fails schema	API error object in valid JSON	Read the error message, likely not retryable blindly
Body parses and validates	Real payload	Proceed

My opinion: do the schema validation at the boundary, in the fetch layer, not scattered across consumers. The counter-argument is that coupling your fetch wrapper to a specific schema makes it less reusable. Fair. Either parameterize the wrapper with an optional model class, or run validation immediately after every safe_json_fetch() call and treat “parsed but wrong shape” as the same class of failure as “didn’t parse” — the recovery logic is identical either way.

Fixing the Root Cause on the Server: Correct Content-Type Headers, Error Handlers, and Redirect Hygiene

Everything above is client-side armor. But if you also control the API — many scraping pipelines hit internal or partner APIs — you should fix the actual disease. An API that returns HTML to API clients is misconfigured, full stop.

FastAPI example — make sure every error path under /api/* returns JSON:

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

@app.exception_handler(404)
async def not_found_handler(request: Request, exc):
    if request.url.path.startswith("/api/"):
        return JSONResponse(
            status_code=404,
            content={"error": "not_found", "path": request.url.path},
        )
    return None  # fall through to default HTML 404 for pages

@app.exception_handler(500)
async def server_error_handler(request: Request, exc):
    if request.url.path.startswith("/api/"):
        return JSONResponse(
            status_code=500,
            content={"error": "internal_error"},
        )
    return None

The nginx side matters just as much, because proxy-level failures bypass your application entirely. The upstream app being down produces the default nginx 502 HTML page unless you override it:

location /api/ {
    proxy_pass http://app_backend;

    proxy_intercept_errors on;

    error_page 502 503 504 = @api_error;
    error_page 404 = @api_not_found;
}

location @api_error {
    default_type application/json;
    return 502 '{"error": "bad_gateway", "hint": "upstream unavailable"}';
}

location @api_not_found {
    default_type application/json;
    return 404 '{"error": "not_found"}';
}

proxy_intercept_errors on is the line most people miss. Without it, nginx passes through whatever the upstream sent — including its HTML error pages. With it, nginx intercepts the error status codes and lets your error_page handlers produce a consistent JSON body.

And the SPA fallback trap: if your frontend router has a catch-all that serves index.html for unmatched routes, exclude /api/ from it. Otherwise every typo’d API path returns your app shell with a 200, which is the single most confusing failure mode in this entire category — the server is healthy, the request “succeeded,” and the payload is a JavaScript bundle tag.

Wrap-up

JSONDecodeError at line 1 column 1 is never a JSON syntax problem. It’s a content negotiation problem: something between your code and the data sent HTML where you expected JSON. The defenses stack cleanly:

1. Sniff Content-Type and the first byte before parsing.
2. Log a truncated, redacted body with status, final URL, and headers.
3. Wrap fetches so failures raise a descriptive exception instead of a cryptic one.
4. Validate the parsed shape, because valid JSON can still be an error object.
5. If you own the server, make every /api/* error path return JSON.

The client-side checks take maybe an hour to retrofit and they convert your worst 2 a.m. alert from a mystery into a one-line diagnosis. If you’re wiring this into a broader pipeline, the retry-routing question comes next — which of these failures are worth retrying, and which need a different fetch strategy — and that’s covered in Client-Side vs Service-Side Retries for Scraping Failures.