Your Scraper Died With Exit Code 137: Fix the OOM Kill

The Failure Signature: Silent Death, Exit 137, and a Death Point That Scales with RAM

Here’s the failure signature. Your crawl worker runs fine for ten minutes. Then the log stops mid-line — no traceback, no exception, no goodbye. The scheduler reports exit code 137. You restart the job and it dies again at almost exactly the same page count. Give it a bigger instance and it gets further before dying anyway.

That combination — silent death, code 137, a death point that scales with available RAM — is the kernel’s OOM killer. No amount of try/except will catch it, and no retry decorator will rescue it, because the process that would run your recovery code never gets to run anything again. This walk-through confirms the kill from system logs, reproduces it on demand under a memory ceiling, profiles the crawl loop to find what’s growing, and replaces the buffering pattern that causes the overwhelming majority of these deaths in scrapers.

Decode 137: Confirm the Kernel OOM Killer Fired Before Touching the Code

Exit codes above 128 follow a convention: the shell reports 128 + N, where N is the signal number that killed the process. Work out N before you change a single line of Python.

$ python worker.py
...normal crawl logs...
$ echo $?
137
$ echo $((137 - 128))
9
$ kill -l 9
KILL

So: 128 + 9 = 137, and signal 9 is SIGKILL. Your worker didn’t crash — something killed it outright. Python raises exceptions to itself; it never sends itself SIGKILL. The usual sender is the kernel’s out-of-memory killer, and it leaves a receipt in the kernel ring buffer. Go read it:

# systems with journald
journalctl -k | grep -i oom

# everywhere else
dmesg -T | grep -i "killed process"

What you’re looking for is the oom-kill report naming your worker and its resident memory at the moment of death:

[ 8431.902313] python3 invoked oom-killer: gfp_mask=0x201dca(GFP_KERNEL), order=0, oom_score_adj=0
[ 8431.902318] CPU: 3 PID: 4721 Comm: python3 Tainted: G
[ 8431.902401] Mem-Info:
[ 8431.902655] active_anon:129536 inactive_anon:65536 isolated_anon:0 ...
[ 8431.903112] Free swap is 0kB
[ 8431.903118] Total swap is 0kB
[ 8431.905512] Out of memory: Killed process 4721 (python3) total-vm:548352kB, anon-rss:522240kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1124kB oom_score_adj:0

Two details matter in that last line. The process name is your worker (python3, PID 4721), and anon-rss:522240kB means it held roughly 510 MB of anonymous memory when it died — right at a 512 MB ceiling. In a container, the wording changes to Memory cgroup out of memory: Killed process ..., and you get a one-command confirmation:

$ docker inspect --format '{{.State.OOMKilled}}' <container>
true

If neither journalctl nor dmesg shows a kill, 137 has other senders — an orchestrator enforcing a timeout with kill -9, a liveness probe gone wrong. But when the process grew steadily toward a limit and died silently, the OOM killer is the odds-on favorite. Confirm it first, then debug.

Why There Is No Traceback: SIGKILL Never Reaches Python’s except or finally

The logs end mid-batch with nothing logged because SIGKILL is handled entirely in the kernel. The process gets no signal handler call, no exception, no stack unwind. It simply stops existing. Watch what does not happen:

import requests

def fetch(url: str) -> str:
    try:
        response = requests.get(url, timeout=30)
        return response.text
    except Exception:
        print("handled!")   # runs for timeouts, connection errors, HTTP errors
    finally:
        print("cleanup!")   # runs on every Python-level exit path

fetch("https://example.com")

Run this against https://example.com and both prints appear on any normal failure. Kill the process with SIGKILL and neither line executes. No finally, no context-manager __exit__, no atexit hook, no destructors.

Event	Catchable in Python	Log output produced	Exit code	Cleanup runs (finally, __exit__, atexit)
SIGTERM	Yes, with a signal handler	Whatever your handler prints	143 (128+15) if unhandled	Only if you handle it
SIGKILL	No	Nothing — the process never runs again	137	Never
Unhandled exception	Yes — this is the normal error path	Full traceback on stderr	1	Yes

This is also why in-process retry logic is useless here. Your tenacity decorator, your max_retries=5, your circuit breaker — they all live inside the process the kernel just removed. The only entities that can observe the death are the kernel (via dmesg) and your supervisor. systemd, for instance, logs Main process exited, code=killed, status=9/KILL, which is where the 137 in your batch report comes from. That’s why the previous section came first: the evidence lives outside your program.

Reproduce the Kill on Demand With a Hard Memory Ceiling

“Woke up dead this morning, works fine now” is the worst kind of bug report. Make it deterministic by pinning the worker to a container or cgroup with a fixed memory budget:

# Docker: cap memory, and cap memory+swap identically so swap can't absorb the ceiling
docker run --memory=512m --memory-swap=512m --rm \
  -v "$PWD":/app -w /app python:3.12-slim python worker.py

# No Docker? Use a systemd scope
systemd-run --user --scope -p MemoryMax=512M python worker.py

Set --memory-swap equal to --memory on purpose. If swap is allowed, the OOM kill turns into a twenty-minute swap-death crawl before the kill lands, and your reproduction gets flaky again. A hard ceiling fails fast and fails the same way every run.

Now instrument the crawl so you can watch memory climb toward the cap. psutil reading the process’s own RSS every 100 pages is enough:

import psutil
import requests

INTERVAL = 100
proc = psutil.Process()
session = requests.Session()
results = []  # the bug, kept in for reproduction

for page in range(1, 50_001):
    url = f"https://store.example.com/products?page={page}"
    resp = session.get(url, timeout=60)
    results.append(resp.text)
    if page % INTERVAL == 0:
        rss = proc.memory_info().rss / (1024 * 1024)
        print(f"page={page:>6}  rss={rss:8.1f} MB", flush=True)

Output under the 512 MB ceiling:

page=   100  rss=    96.4 MB
page=   200  rss=   181.9 MB
page=   300  rss=   267.5 MB
page=   400  rss=   352.8 MB
page=   500  rss=   438.2 MB
Killed

A straight line, roughly 0.85 MB per page, ending at the cap. You have just converted an intermittent production incident into a bug you can trigger in ninety seconds. Keep this harness — it becomes your verification loop later.

Profile the Crawl Loop: Trace Which Objects Own the Growth

You could guess from the linear slope that something accumulates one-page-at-a-time. Don’t guess. tracemalloc attributes retained memory to exact source lines:

import tracemalloc

tracemalloc.start()
prev = None

for page in range(1, 2_001):  # shorter run — tracemalloc adds real overhead
    url = f"https://store.example.com/products?page={page}"
    resp = session.get(url, timeout=60)
    results.append(resp.text)

    if page % 500 == 0:
        snap = tracemalloc.take_snapshot()
        if prev is not None:
            print(f"--- page {page} ---")
            for stat in snap.compare_to(prev, "lineno")[:5]:
                print(stat)
        prev = snap

Sample output:

--- page 2000 ---
worker.py:19: size=1.7 GiB (+438.5 MiB), count=2000 (+500), average=884 KiB
/usr/lib/python3.12/requests/models.py:384: size=2.1 MiB (+0.2 MiB), count=6180 (+1545)
worker.py:12: size=0.6 MiB (+0.0 MiB), count=38 (+1)

Read that first line as a signed confession: the results.append(...) at line 19 owns 1.7 GiB and grows by ~438 MiB every 500 pages — about 884 KiB per page, matching your body size. Everything else is noise.

Two caveats. Tracemalloc adds overhead (often roughly doubling the tracked allocations’ footprint), so run it on a shorter, uncapped crawl rather than inside your 512 MB reproduction. And it reports Python-level allocations — it will not show you memory held by C extensions. For scrapers, the culprit is almost always Python-side anyway, and this snapshot proves it.

The Usual Culprit: results.append(response.text) Across 50,000 Pages

Here is the antipattern in its natural habitat. Fetch every page, keep every body in a list, and write everything at the end:

import requests

session = requests.Session()
results = []

for page in range(1, 50_001):
    resp = session.get(
        f"https://store.example.com/products?page={page}", timeout=60
    )
    results.append(resp.text)

# one final bulk write... which is never reached
with open("products.json", "w", encoding="utf-8") as fh:
    json.dump(results, fh)

Run the arithmetic against a 512 MB limit, assuming ~900 KB product pages:

Pages buffered	Retained bodies (900 KB each)	Against the 512 MB cap
1	0.9 MB	Irrelevant
100	~90 MB	Fine — RSS ~150 MB with interpreter overhead
500	~450 MB	RSS ~505 MB. At the ceiling.
~508	~457 MB	OOM kill lands here
50,000	~45 GB	Would need roughly 88x the cap

The kill lands near page 500 rather than the naive 568 because CPython, requests, TLS buffers, and your logging setup claim a baseline of ~55 MB before the first page. Two subtleties make it slightly worse than the table suggests. First, response.text requires response.content to exist, so during the decode both the bytes and the string are alive at once. Second, glibc does not eagerly return freed arenas to the OS — even transient spikes leave RSS near its high-water mark, which is why del results halfway through the loop saves you far less than you’d hope.

Bumping the container to 2 GB is not a fix; it relocates the kill to around page 2,200. The list grows without bound, so any fixed ceiling loses eventually.

Fix It: Stream Each Body to Disk Instead of Holding It in Memory

Replace whole-body retention with per-page persistence. Stream the response in chunks, feed the chunks to a small parser that keeps only the fields you want, write one row per page, and let each page’s footprint return to baseline before the next request:

import gzip
import json
import os
import requests
from html.parser import HTMLParser

RAW_DIR = "raw"
CHUNK = 64 * 1024

class ProductParser(HTMLParser):
    """Keeps a few fields; never holds the full document."""
    def __init__(self):
        super().__init__()
        self._in_h1 = False
        self.title = ""
        self.price = None

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if tag == "h1":
            self._in_h1 = True
        if tag == "meta" and attrs.get("property") == "product:price:amount":
            self.price = attrs.get("content")

    def handle_endtag(self, tag):
        if tag == "h1":
            self._in_h1 = False

    def handle_data(self, data):
        if self._in_h1:
            self.title += data

def crawl(pages, out_path="products.ndjson"):
    os.makedirs(RAW_DIR, exist_ok=True)
    session = requests.Session()
    with open(out_path, "a", encoding="utf-8") as rows:
        for page in pages:
            url = f"https://store.example.com/products?page={page}"
            parser = ProductParser()
            raw_path = os.path.join(RAW_DIR, f"page-{page:06d}.html.gz")
            with session.get(url, stream=True, timeout=60) as resp:
                resp.raise_for_status()
                with gzip.open(raw_path, "wb") as raw:
                    for chunk in resp.iter_content(chunk_size=CHUNK):
                        raw.write(chunk)
                        parser.feed(chunk.decode("utf-8", errors="replace"))
            rows.write(json.dumps(
                {"page": page, "title": parser.title.strip(), "price": parser.price}
            ) + "\n")

crawl(range(1, 50_001))

Peak memory per page is now one 64 KB chunk plus parser state — call it a few hundred KB instead of 900 KB retained forever. The gzip copy of each raw body costs disk, not RAM, and buys you the ability to re-parse later without re-crawling, a trade-off worth making; I wrote up the full reasoning in Parse at Fetch Time vs Store Raw HTML. The append-per-page NDJSON layout has a second payoff: a restart resumes near where it died instead of from zero, which pairs well with a checkpointed pipeline.

One caveat and one honest cost. The caveat: chunk boundaries can split a UTF-8 sequence, so errors="replace" is fine for field scraping but decode from the archived file when you need exact bytes. The cost: you now do small writes on every page. Batch your flush() every N rows if I/O shows up in your profile — but flush often enough that a crash costs minutes, not hours.

Rerun the psutil logger from the reproduction section against this version. The same 50,000-page batch over store.example.com shows RSS sitting near 92 MB from page 100 to page 49,900, flat the whole way.

A related note, because it bites people who think they outsourced the problem: offloading rendering to a scraping API does not change this math. If you call FineData, or any other endpoint, and keep every returned JSON envelope in a list until the batch finishes, the kill happens on your side of the wire. The remote service rendered the page; your process still buffered 50,000 copies of it.

Framework Case: Cap Concurrency and Flush Items in a Scrapy Pipeline

Scrapy dies the same death with a different mechanism. Instead of one giant list, it is dozens of in-flight response bodies plus items stashed on the spider instance. Same kernel, same 137.

# settings.py
CONCURRENT_REQUESTS = 8
CONCURRENT_REQUESTS_PER_DOMAIN = 4
MEMUSAGE_ENABLED = True
MEMUSAGE_LIMIT_MB = 512
ITEM_PIPELINES = {"myproject.pipelines.JsonlPipeline": 100}

The pipeline writes each item the moment it arrives instead of collecting a list:

# pipelines.py
import json

class JsonlPipeline:
    def open_spider(self, spider):
        self.file = open("products.jl", "a", encoding="utf-8")

    def close_spider(self, spider):
        self.file.close()

    def process_item(self, item, spider):
        self.file.write(json.dumps(item, default=str) + "\n")
        self.file.flush()
        return item

And in the spider, yield items — never self.products.append(item). Yielding hands ownership to the pipeline; stashing on self keeps every item alive until the spider closes.

Results from the same 512 MB capped harness, same crawl:

CONCURRENT_REQUESTS	Peak RSS	Outcome
32	~700 MB by page ~400	OOM kill, exit 137
16	~380 MB	Completed, uncomfortably close to the ceiling
8	~230 MB	Completed, exit 0

MEMUSAGE_LIMIT_MB deserves a mention: when the limit is reached, the MemoryUsage extension closes the spider cleanly instead of letting the kernel shoot it. You get an exit you control rather than SIGKILL — which turns a mystery into a log line.

An opinion you may disagree with: almost nobody needs CONCURRENT_REQUESTS=32 against a single domain. At 32 in-flight requests you are holding ~28 MB of raw HTML at all times before counting decoded copies, Scrapy’s caches, and your item objects — and you are hammering one origin hard enough that throttling responses are near-certain. I’d rather run 8 concurrent, finish in a slower wall time, and never see the OOM killer. Throughput that gets your worker killed mid-batch isn’t throughput; it’s a rerun you’ll pay for twice.

Verify the Fix: Peak RSS and Exit Status Before vs After

Never call a memory fix done without numbers. GNU time gives you peak RSS and exit status in one shot (use /usr/bin/time, not the shell builtin, which silently ignores -v):

/usr/bin/time -v python worker.py 2> time.log
grep -E "Maximum resident|Exit status" time.log
#   Maximum resident set size (kbytes): 92416
#   Exit status: 0

Full comparison over the store.example.com crawl, all inside the same 512 MB ceiling:

Build	Peak RSS	Pages completed	Exit code
Buffered list, single write at end	512 MB (cap hit)	~508 of 50,000	137
Streamed chunks, per-page NDJSON + gzip	92 MB	50,000	0
Scrapy, 8 concurrent, flush-per-item pipeline	231 MB	50,000	0

Then check the source of the original evidence — the kernel log — for new kills since the change:

journalctl -k --since "1 hour ago" | grep -i "killed process" | grep python3 \
  || echo "no new oom kills"

Empty output plus a zero exit status under the same capped container that killed the old build is proof. Keep the capped container around and run every future memory-related change through it once; it takes two minutes and catches regressions before production does.

Wrap-Up

Exit code 137 is not a Python error. It’s a receipt from the kernel stating that your worker crossed a memory line and was removed without ceremony. The debugging order that works: confirm the OOM kill in journalctl/dmesg, reproduce it under a fixed ceiling, attribute the growth with tracemalloc, then stop buffering whole bodies. Stream responses in chunks, persist per page or per item, and cap concurrency so in-flight bodies can’t stack up.

The uncomfortable version of this advice: if your fix was raising the memory limit, you didn’t fix anything — you moved the death page from 508 to 2,200 and bought a rerun. Fix the retention pattern instead, verify peak RSS with /usr/bin/time -v, and the OOM killer stops being part of your on-call story.