Gzip Says Unexpected EOF: Fix Truncated Page Bodies

Why “unexpected end of file” Means Your Gzip Stream Is Incomplete, Not Corrupt

The traceback shows up in your pipeline logs at 3 a.m. and looks like this:

import gzip

# raw_body came from a fetch of https://store.example.com/products/widget-42
# and was cut off partway through the compressed stream
with open("truncated_body.gz", "rb") as f:
    raw_body = f.read()

gzip.decompress(raw_body)

Traceback (most recent call last):
  File "pipeline.py", line 118, in fetch_and_parse
    html = gzip.decompress(raw_body)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "gzip.py", line 342, in decompress
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was encountered

Depending on which library touched it, the same failure surfaces as zlib.error: Error -5 while writing data to decompression stream, EOFError: Compressed file ended before the end-of-stream marker was encountered, or — if you’re on a lower level — OSError: unexpected end of file. All of these say the same thing: the decompressor ran out of input before the gzip stream told it the data was finished.

A gzip stream has a defined structure: a header, a series of DEFLATE blocks, a final block flag, and an eight-byte trailer containing a CRC32 checksum and the uncompressed size. The decompressor needs to see that trailer to know the stream is complete. “Unexpected end of file” means the bytes stopped arriving before the trailer did. The data you have isn’t garbage — it’s a valid prefix of a valid stream that simply never arrived in full.

That distinction matters, because it points the investigation outward. You don’t have an encoding bug. You have a transport problem. Something between the origin server and your process dropped bytes, and the gzip layer was just the first place honest enough to notice.

Distinguish it from the other decompression failures you’ll see, because the fixes are completely different:

Failure signature	What it actually means	Typical root cause
EOFError: Compressed file ended before the end-of-stream marker / unexpected end of file	Stream is a valid prefix but missing its tail	Connection cut mid-body, response size cap, truncated read
BadGzipFile: CRC check failed / gzip: invalid compressed data--crc error	Stream finished but the checksum doesn’t match the data	Bytes were altered in transit, corrupted proxy cache, memory corruption
zlib.error: Error -3 while decompressing data: incorrect header check	The first bytes aren’t a gzip header at all	Double-encoded body, server ignored Accept-Encoding, HTML error page delivered with Content-Encoding: gzip anyway

If you see the third one, read your 200 OK responses are actually block pages before anything else — that’s a different disease. This post is about the first signature, and the rest of it is a systematic hunt for the missing bytes.

Tracing the Truncation: Reading Content-Length, Transfer-Encoding, and Raw Byte Counts

Before you can fix anything, you need to know which side of the connection lost the bytes. The server declared a size; did it deliver? Instrument the fetch so the mismatch becomes a number instead of a mystery.

import requests

def fetch_verbose(url: str) -> tuple[bytes, dict]:
    with requests.get(
        url,
        headers={"Accept-Encoding": "gzip"},
        timeout=(5, 30),
        stream=True,
    ) as resp:
        meta = {
            "status": resp.status_code,
            "content_encoding": resp.headers.get("Content-Encoding"),
            "content_length": resp.headers.get("Content-Length"),
            "transfer_encoding": resp.headers.get("Transfer-Encoding"),
            "connection": resp.headers.get("Connection"),
        }
        chunks = []
        for chunk in resp.iter_content(chunk_size=8192):
            chunks.append(chunk)
        raw = b"".join(chunks)

        declared = resp.headers.get("Content-Length")
        if declared is not None:
            meta["declared_bytes"] = int(declared)
            meta["received_bytes"] = len(raw)
            meta["shortfall"] = int(declared) - len(raw)
        else:
            meta["declared_bytes"] = None
            meta["received_bytes"] = len(raw)
            meta["shortfall"] = None
        return raw, meta

raw, meta = fetch_verbose("https://store.example.com/products/widget-42")
print(meta)

Run this against the URL that’s failing and read the output:

{
  'status': 200,
  'content_encoding': 'gzip',
  'content_length': '45000',
  'transfer_encoding': None,
  'connection': 'keep-alive',
  'declared_bytes': 45000,
  'received_bytes': 31802,
  'shortfall': 13198
}

That’s your answer in four lines. The server promised 45,000 bytes and delivered 31,802. The connection closed early, and the gzip stream died 13,198 bytes short of its trailer.

Three diagnostic branches fall out of this output:

• Content-Length present, received bytes match, decompression still fails. The body arrived whole. Something mangled it between the socket and your decompressor — a middleware layer, a double Content-Encoding header, or a caching layer that re-compressed a partial body. Rare, but it happens.
• Content-Length present, received bytes short. The connection was cut mid-body. This is the common case and the one the rest of this post attacks.
• No Content-Length, Transfer-Encoding: chunked instead. The server never declared a size, so you can’t detect truncation from headers alone. You find out only when the decompressor hits the missing trailer — or worse, when the chunked terminator chunk (0\r\n\r\n) never arrives and your HTTP client times out instead. Chunked responses need the decompressor-based detection covered below.

One subtlety: requests and httpx transparently decompress the body when Content-Encoding: gzip is set. If you’re seeing gzip errors, either you’re reading resp.raw directly, or the server mislabeled the body. To get the raw compressed bytes for instrumentation like the above, read from the raw stream without the decode helpers, or set headers={"Accept-Encoding": "gzip"} and use resp.raw.read() with decode_content=False semantics. The point of this instrumentation is to count wire bytes, not decoded bytes.

The Three Usual Culprits: Premature Connection Closes, Response Size Limits, and Chunked-Encoding Bugs

Once you’ve confirmed a byte shortfall, there are three causes that cover almost every case I’ve debugged.

Culprit 1: premature connection close on a keep-alive connection. You’re reusing a pooled connection. The server (or a load balancer in front of it) has a keep-alive timeout of, say, 60 seconds, but its clock and yours drift, or the pool handed you a connection the server closed a millisecond ago. You send a request down a dead socket, the server’s FIN arrives mid-response, and your client reads a partial body before the socket reports closure. This is the classic race. It’s also why truncation often correlates with idle periods between batches: the longer your worker sat quiet, the staler the pooled connection.

Culprit 2: a response size cap on an intermediary. Reverse proxies don’t always pass big bodies through untouched. A config like this on the origin serving store.example.com:

# nginx reverse proxy in front of the origin app
proxy_buffering on;
proxy_buffer_size 16k;
proxy_buffers 8 32k;
# the dangerous one — caps how much nginx will read from the upstream:
proxy_max_body_size 2m;   # applies to request bodies, but note the pattern
# and on the response path, an aggressive timeout:
proxy_read_timeout 15s;

A proxy_read_timeout that’s shorter than the origin’s page-render time kills the upstream connection mid-body; nginx then serves whatever it buffered, or closes the client connection with a partial response. Some WAFs and CDN edge configs do the same with hard response-size ceilings. The telltale: truncation happens at a consistent byte count for the same URL, and it reproduces on every retry.

Culprit 3: chunked-encoding mishandling. Either your client’s chunked parser has a bug (rare with requests/httpx, common with hand-rolled asyncio HTTP), or an intermediary rewrote the response from Content-Length to chunked and got the framing wrong. The signature here is decompression failing at a different byte offset on each attempt, or the client hanging waiting for the terminating chunk.

Culprit	Content-Length present?	Declared vs received	Byte count of truncation	Reproducible?
Premature connection close	Yes	Received < declared	Varies per attempt	Intermittent, correlates with connection age / idle time
Intermediary size cap or read timeout	Yes, usually	Received < declared, often a buffer-sized multiple	Same byte count for the same URL	Yes, every time
Chunked-encoding bug	No (chunked)	N/A — no declared size	Varies; sometimes hangs on terminator	Intermittent, worse under concurrency

The “consistent byte count” column is the fastest triage you have. Retry the same URL five times. If it truncates at the same offset every time, stop blaming your network and look upstream. If the offset moves, suspect your own connection pool first.

Fixing the Client: Streaming Decompression with Incremental zlib Instead of One-Shot Reads

The naive pattern is body = resp.read() followed by gzip.decompress(body). It works until it doesn’t, and when it doesn’t, you get the opaque EOFError with no partial data and no early warning. The fix is to decompress incrementally as bytes arrive. This gives you three things: early detection (you know the stream is broken the moment it ends, not after a full read), graceful accounting (you know exactly how many bytes arrived), and the option to keep the valid prefix for debugging.

import zlib

class TruncatedBodyError(Exception):
    def __init__(self, received: int, host: str):
        self.received = received
        self.host = host
        super().__init__(
            f"gzip stream from {host} ended early after {received} wire bytes"
        )

def stream_decompress(chunks):
    """Incrementally decompress a gzip stream from an iterable of byte chunks.

    Raises TruncatedBodyError if the stream ends before the gzip trailer.
    """
    # 16 + MAX_WBITS tells zlib to expect a gzip wrapper (header + trailer)
    # instead of a raw DEFLATE stream.
    dec = zlib.decompressobj(16 + zlib.MAX_WBITS)
    out = []
    wire_bytes = 0
    for chunk in chunks:
        wire_bytes += len(chunk)
        out.append(dec.decompress(chunk))

    # The critical check: if the stream was complete, the decompressor
    # reports eof=True and unconsumed_tail is empty. If eof is False,
    # the gzip trailer never arrived.
    if not dec.eof:
        raise TruncatedBodyError(wire_bytes, "store.example.com")

    return b"".join(out)

dec.eof is the whole trick. The zlib decompressor object knows whether it saw the final block flag and the trailer. One-shot gzip.decompress() converts that fact into a cryptic EOFError; the incremental object lets you convert it into an exception that carries the wire byte count and the host, which is exactly what your retry logic and your monitoring need.

Now wrap the fetch in a retry. Truncation from stale connections is transient — a fresh connection usually gets the full body — so a bounded retry with exponential backoff is the right first response:

import time
import requests

def fetch_with_retry(url: str, max_attempts: int = 4):
    last_err = None
    for attempt in range(max_attempts):
        # fresh session per attempt: forces a new TCP connection,
        # sidestepping the stale keep-alive race entirely
        with requests.Session() as s:
            s.headers["Accept-Encoding"] = "gzip"
            with s.get(url, timeout=(5, 30), stream=True) as resp:
                resp.raise_for_status()
                chunks = [c for c in resp.iter_content(chunk_size=8192)]
        try:
            return stream_decompress(chunks)
        except TruncatedBodyError as e:
            last_err = e
            backoff = min(2 ** attempt, 8)
            time.sleep(backoff + (attempt * 0.1))
    raise last_err

An honest trade-off note: creating a fresh session per attempt costs you a TCP plus TLS handshake, maybe 100-300 ms per retry. For a pipeline that only retries on truncation — an event that should be rare — that cost is noise. If truncation is frequent enough that the handshakes matter, the retries are masking a real upstream problem, and you should skip to the monitoring section. Retries are a bandage, not a treatment. I’d rather see a pipeline with zero retries and loud alerts than one silently retrying 30% of its fetches.

For more on where retries should live, see client-side vs service-side retries for scraping failures — the same logic applies to truncated bodies as to 429s.

Hardening the Fetch: Timeouts, Connection Reuse Hygiene, and Verifying Byte Counts

Most truncation never reaches the decompressor if you configure the fetch correctly. Three habits cover the majority of it.

Habit 1: explicit two-part timeouts. The single-value timeout=30 in requests sets both connect and read timeouts to 30 seconds. That’s usually fine, but the read timeout deserves its own thought: it’s the maximum gap between bytes arriving, not total transfer time. A slow-drip response from an overloaded origin can legitimately take 90 seconds on a 200 KB page. Set them separately and deliberately:

import httpx

# connect timeout tight (a healthy server completes the handshake fast),
# read timeout generous (big compressed pages can trickle)
client = httpx.Client(
    timeout=httpx.Timeout(connect=5.0, read=60.0, write=10.0, pool=5.0),
    limits=httpx.Limits(max_keepalive_connections=10, keepalive_expiry=30.0),
)

def fetch(client: httpx.Client, url: str) -> bytes:
    with client.stream("GET", url) as resp:
        resp.raise_for_status()
        declared = resp.headers.get("Content-Length")
        chunks = []
        for chunk in resp.iter_bytes():
            chunks.append(chunk)
    raw = b"".join(chunks)

    if declared is not None and len(raw) != int(declared):
        raise TruncatedBodyError(len(raw), "store.example.com")

    return stream_decompress(chunks)

Habit 2: keepalive hygiene. The keepalive_expiry=30.0 line above matters more than it looks. httpx will close pooled connections after 30 seconds of idleness, which should be comfortably shorter than the server’s own keep-alive window. If your client holds connections longer than the server does, you’ll periodically pull a dead connection from the pool and eat a truncated response. If you can’t control the server’s window, be conservative: retire connections early and pay the handshake. For batch jobs, an even simpler policy works — one httpx.Client per batch, closed when the batch ends, so no connection survives an idle period long enough to go stale.

Habit 3: verify byte counts before decompressing. The explicit check in the code above — comparing len(raw) to the declared Content-Length — turns “mysterious EOFError deep in gzip” into “declared 45,000, got 31,802” at the fetch boundary. Fail fast, with the numbers in the error message:

def verify_length(raw: bytes, declared: str | None, host: str) -> bytes:
    if declared is None:
        return raw  # chunked: no contract to verify, rely on decompressor
    expected = int(declared)
    if len(raw) != expected:
        raise TruncatedBodyError(
            received=len(raw), host=host
        )  # carries both numbers for the log
    return raw

Is this check redundant with dec.eof? Almost. The decompressor catches truncation on every response, including chunked ones. The length check catches it earlier, with cleaner attribution, and it also catches the opposite failure — receiving more bytes than declared, which usually means an intermediary appended garbage or you’re reading two responses off one socket. Cheap insurance.

When the Server Is at Fault: Handling Origin-Side Truncation and Partial-Content Workarounds

Sometimes the client is innocent. The origin is overloaded, its reverse proxy has a read timeout shorter than its own render time, and it genuinely serves truncated bodies to everyone. Retrying against the same broken path just burns tokens and time. Two workarounds exist, with different trade-offs.

Range requests. If the server supports byte ranges (check Accept-Ranges: bytes in the response headers), you can fetch the body in segments and verify each one. This is mostly a diagnostic tool and a recovery path for large bodies that consistently truncate at the same offset:

import requests

def fetch_by_ranges(url: str, total: int, segment: int = 32768) -> bytes:
    parts = []
    offset = 0
    while offset < total:
        end = min(offset + segment - 1, total - 1)
        r = requests.get(
            url,
            headers={
                "Accept-Encoding": "gzip",
                "Range": f"bytes={offset}-{end}",
            },
            timeout=(5, 30),
        )
        if r.status_code not in (200, 206):
            raise RuntimeError(f"range fetch failed: {r.status_code}")
        parts.append(r.content)
        if len(r.content) < (end - offset + 1):
            raise TruncatedBodyError(len(r.content), "store.example.com")
        offset = end + 1
    return b"".join(parts)

# usage: first fetch truncated at 31,802 of a declared 45,000 bytes
body = fetch_by_ranges("https://store.example.com/products/widget-42", total=45000)
html = stream_decompress([body])

Be honest with yourself about when this is worth it. Range-request stitching triples your request count and only works if the server supports ranges on dynamically generated responses — many don’t, and some CDNs strip the Range header for uncached content. I’d use it for a one-off recovery of a critical page, not as a pipeline pattern. If a host needs range-stitching routinely, it needs to be on your quarantine list instead.

Identity encoding as a diagnostic. Request the body uncompressed and compare sizes. This isolates whether gzip is the failure point or just the messenger:

r = requests.get(
    "https://store.example.com/products/widget-42",
    headers={"Accept-Encoding": "identity"},
    timeout=(5, 30),
)
# If this returns the full page, the truncation is specific to the
# compressed response path — likely a proxy buffering or size-cap issue.
# If this also truncates, the origin itself is dropping the connection.

The comparison is decisive. Full uncompressed body plus truncated compressed body means an intermediary is mishandling the compressed response — and yes, this happens; some proxy configs buffer compressed responses differently than plain ones. Truncation in both modes means the origin or its connection handling is at fault, and no client-side fix will save you.

A pragmatic alternative when a specific origin keeps truncating: route the fetch through a scraping API and let the service-side retry machinery absorb it. A single call to the sync scrape endpoint with default retry behavior offloads the connection management entirely:

import requests

resp = requests.post(
    "https://api.finedata.ai/api/v1/scrape",
    headers={"Authorization": "Bearer fd_your_api_key"},
    json={
        "url": "https://store.example.com/products/widget-42",
        "formats": ["rawHtml"],
        "auto_retry": True,
        "max_retries": 5,
        "timeout": 120,
    },
    timeout=300,
)

The service retries server-side on incomplete fetches, so a truncated first attempt doesn’t surface in your pipeline at all. Whether that’s cheaper than fixing your own connection hygiene depends on volume — see web scraping API vs DIY: total cost of ownership — but for a small set of stubborn hosts it’s a reasonable escape hatch.

Making It Hold Up at Scale: Monitoring Truncation Rates and Circuit-Breaking Bad Hosts

Everything above fixes individual fetches. At scale, the question changes: which hosts truncate, how often, and when do you stop trying? You need per-host truncation telemetry, because a 2% truncation rate spread across your fleet is noise, while 2% concentrated on one host is a signal.

Log the contract on every fetch:

import logging
import time

logger = logging.getLogger("fetch.telemetry")

def log_fetch_telemetry(host: str, declared, received: int, truncated: bool):
    logger.info(
        "fetch_telemetry",
        extra={
            "host": host,
            "bytes_expected": declared if declared is not None else -1,
            "bytes_received": received,
            "truncated": int(truncated),
            "ts": time.time(),
        }
    )

# in your fetch path:
# log_fetch_telemetry("store.example.com", 45000, 31802, True)

If you run Prometheus, expose it as counters instead — fetch_truncations_total{host=...} and fetches_total{host=...} — and compute the rate per host. The raw log lines work fine for smaller operations; just make sure the truncation flag is a first-class field you can aggregate, not a string you grep for at incident time.

Then set thresholds with graduated responses. My opinionated position: alert on rate, not count, and make the automated response proportional. A host that truncates once at 3 a.m. should not page anyone.

Condition (per host, 15-min window)	Response
Truncation rate below 0.5%	Log only; retries absorb it
0.5% to 2%	Warning alert; check for a consistent truncation offset (points to an upstream size cap)
2% to 5%	Auto-reduce concurrency to that host to 1; suspect your own connection pool pressure
Above 5% for two consecutive windows	Quarantine: pull host from rotation, route through the scraping API or range-stitching path, file it for manual triage

The concurrency-reduction step deserves a note. High truncation rates that appear only under load are frequently self-inflicted — you’re opening more connections than the server’s keep-alive table tolerates, and it starts dropping them mid-response. Before quarantining a host, always test it single-threaded. If truncation vanishes at concurrency 1, the host isn’t broken; your throughput target for it is.

For the quarantine mechanics — how to trip the breaker, how to probe for recovery, and how to avoid flapping — the pattern in trip a circuit breaker when your scrape target degrades maps directly onto truncated-body detection. The trigger condition changes; the state machine doesn’t.

Wrap-Up

“Gzip says unexpected EOF” is never a compression problem. It’s a transport problem wearing a compression costume. The chain of reasoning is short: the error means the stream lost its trailer, the trailer is missing because bytes never arrived, and the bytes never arrived because a connection died, an intermediary capped the response, or your chunked handling broke.

The fixes stack in order of leverage. Instrument first — declared versus received byte counts turn a mystery into a number. Then harden the fetch: explicit connect/read timeouts, keepalive expiry shorter than the server’s, and length verification before decompression. Swap one-shot gzip.decompress() for an incremental zlib.decompressobj(16 + zlib.MAX_WBITS) so truncation detection is precise and carries its own evidence. Retry with fresh connections for the transient cases, but watch the retry rate — retries that fire constantly are hiding an upstream fault you should be alerting on instead.

And build the operational layer before you need it. Per-host truncation counters, rate-based thresholds, and a quarantine path for hosts that consistently serve partial bodies. The decompressor already knows when a stream is incomplete. All you have to do is listen to it at fleet scale.