Certificate Verify Failed: Fix SSL Errors in Web Scrapers

Decode the Exact Failure: Which Verification Step Did Your Scraper Trip On?

It works on your laptop. You push the pipeline, CI spins up, and ninety seconds later the run is red with a wall of traceback ending in SSLError. The scraper touched nothing unusual — same URL, same code — but the container it landed in has a different idea about what a trustworthy certificate looks like.

Here is the failure in full, from a requests-based scraper fetching a product page:

Traceback (most recent call last):
  File "scraper.py", line 41, in <module>
    resp = requests.get("https://example.com/products", timeout=30)
  File ".../requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File ".../requests/adapters.py", line 486, in send
    resp = conn.urlopen(...)
  File ".../urllib3/connectionpool.py", line 878, in urlopen
    return self.urlopen(method, url, body, headers, retries, ...)
  File ".../urllib3/connection.py", line 418, in connect
    self.sock = ssl_wrap_socket(...)          # 1. urllib3 hands the raw socket to ssl
  File ".../ssl.py", line 1345, in do_handshake
    self._sslobj.do_handshake()               # 2. OpenSSL runs the TLS handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1020)

Two things to internalize from this stack. First, requests verifies nothing itself — it delegates to urllib3, which delegates to Python’s ssl module, which delegates to OpenSSL. Second, and this is the part people miss: OpenSSL runs four independent checks during verification, and the traceback only reports the first one that fails. The tail of that last line — everything after certificate verify failed: — is your actual error message. Everything above it is plumbing.

Map the tail to the check:

#	Check	Question OpenSSL asks	Error string when it fails
1	Chain of trust	Can I link the leaf cert to a known root, via intermediates?	unable to get local issuer certificate
2	Trust anchor	Is the top of that chain actually in my local store?	self signed certificate in certificate chain
3	Validity window	Is “now” between notBefore and notAfter?	certificate has expired, certificate is not yet valid
4	Hostname match	Does the cert’s SAN list the host I asked for?	hostname mismatch, certificate is not valid for 'example.com'

unable to get local issuer certificate — the most common variant by far — means check 1 or 2 failed: either the server didn’t hand you enough chain to work with, or your trust store doesn’t contain the root that chain ends at. Those are two different problems with two different fixes, which is why guessing at this point wastes an afternoon.

Diagnose Before Patching: Inspect the Live Certificate Chain with openssl

Before you touch a line of Python, look at what the server actually presents. One command answers most of it:

openssl s_client -connect example.com:443 \
  -servername example.com -showcerts </dev/null

Always pass -servername. Without SNI, a virtual-hosted server returns its default certificate, which frequently doesn’t match the hostname you actually wanted — that alone produces a hostname mismatch that has nothing to do with your code. Annotated output, trimmed to what matters:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1   <-- root
depth=1 C = US, O = Let's Encrypt, CN = R11                              <-- intermediate
depth=0 CN = example.com                                                 <-- leaf, served by the site
---
Certificate chain
 0 s:CN = example.com                            <-- what the cert claims to be
   i:C = US, O = Let's Encrypt, CN = R11          <-- who signed it
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
...
Verify return code: 0 (ok)

Read the s:/i: pairs as subject/issuer. Each certificate’s issuer must equal the next certificate’s subject, walking upward, until you hit something your trust store recognizes. The Verify return code at the bottom is OpenSSL’s verdict using its own defaults — not your Python process’s — so treat it as a second opinion:

• 0 (ok) — the chain is fine; your problem is inside the Python environment, not the server.
• 20 (unable to get local issuer certificate) — the chain is incomplete or the root is missing from the store OpenSSL used.
• 19 (self-signed certificate in chain) — something in the chain isn’t a public CA. On an office network, that usually means interception.
• 10 (certificate has expired) — validity window failure.

That last one deserves its own routine, because the validity check compares the certificate against your system clock:

date -u
# compare against:
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=...   must be in the past
# notAfter=...    must be in the future

Fresh cloud VMs and containers without NTP are a classic source of certificate is not yet valid. If date -u shows a boot-time default instead of real time, your scraper was never broken. Fix the clock, not the code.

Stale CA Bundles: Why Your Scraper Works Locally but Fails in Docker and CI

Python’s requests and httpx do not use your operating system’s trust store by default. They use certifi, a CA bundle that ships inside the wheel. Pin your dependencies, bake the image, forget about it — and the bundle inside that image is frozen at build time. CAs rotate intermediates and roots on their own schedule; when a site you scrape switches to a chain your snapshot doesn’t know, every request starts failing with unable to get local issuer certificate. The site is healthy. Your trust store is the fossil.

Measure the gap before assuming anything. This one-liner works identically on your laptop and in the CI container:

python -c "import certifi; print(certifi.__version__, certifi.where())"

If the versions differ, you’ve found it. Fix the image, not the script:

FROM python:3.12-slim

# Refresh the OS-level store too: pip tools, curl, and anything
# shelling out to the system still rely on it.
RUN apt-get update \
 && apt-get install -y --no-install-recommends ca-certificates \
 && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir --upgrade certifi

Here is an opinion that gets pushback in code review: do not hard-pin certifi in requirements.txt the way you pin application code. A CA bundle is data that decays, not an interface you depend on — pinning it trades reproducibility you don’t need for a trust store that goes stale on a schedule you don’t control. Pin your frameworks; let the bundle move, and schedule image rebuilds so “latest” doesn’t quietly mean “eighteen months old.”

One adjacent classic: a python.org installer on macOS where Install Certificates.command was never run. The interpreter links against an OpenSSL with no roots at all, and every HTTPS call fails. If a single machine misbehaves while containers pass, that is the first place I look.

When the fetching itself is delegated to a scraping API rather than your own runtime, this entire failure class stops being yours — FineData and similar services terminate TLS to the target upstream, on infrastructure whose trust store someone else keeps current. For pipelines where Docker image hygiene is shaky, that is a legitimate reason to offload:

curl -X POST https://api.finedata.ai/api/v1/async/scrape \
  -H "Authorization: Bearer fd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/", "formats": ["markdown"]}'

Your container’s certifi version is irrelevant to that request’s TLS handshake with the target site.

TLS Inspection: When a Corporate Proxy Re-Signs Every Certificate You Request

Sometimes the bundle isn’t stale and the server isn’t broken — your network is rewriting the conversation. Corporate security appliances intercept outbound TLS, present their own certificate to your client, and open a separate connection upstream. To the scraper, the “example.com” certificate is now signed by the security team, not by a public CA. openssl s_client run from two networks makes this unmistakable:

Field	From home network	Behind the corporate proxy
Leaf subject	CN = example.com	CN = example.com (re-created)
Leaf issuer	C=US, O=Let's Encrypt, CN=R11	O=Example Corp Security, CN=Example Proxy CA
Chain top	ISRG Root X1 (public root)	Example Corp Root CA (private root)
Verify return code	0 (ok)	19 (self-signed certificate in chain)

Same URL, same day, different issuer. That is interception by design, and the correct response is to trust the corporate root on corporate hardware — not to disable verification. Get the root certificate from IT (they publish it for exactly this purpose), then merge it into a copy of the standard bundle:

import certifi
from pathlib import Path

merged = (
    Path(certifi.where()).read_bytes()      # public roots
    + b"\n"
    + Path("corp-root-ca.pem").read_bytes() # the proxy's root
)
Path("merged-ca.pem").write_bytes(merged)

export REQUESTS_CA_BUNDLE="$PWD/merged-ca.pem"

Two boundaries matter. Never commit merged-ca.pem to a shared repository — that root is for company machines, and leaking it into production images makes the proxy’s CA trusted where no proxy exists, which is pure downside. And be honest about what you accepted: trusting that root means the appliance can decrypt anything, which is the employer’s stated policy, not a bug you found. If you disagree with the policy, that argument happens with IT. Fighting it in code with verify=False is the one move that’s strictly worse than either outcome.

Missing Intermediates: When the Server, Not Your Scraper, Is Broken

Every so often the site itself is the defect. A misconfigured nginx serves only its leaf certificate and forgets the intermediate, and here is the maddening part: Chrome shows a green padlock anyway, because browsers cache intermediates from earlier handshakes and some will even fetch the missing cert from the CA’s AIA endpoint. OpenSSL does neither. Your scraper fails while everyone with a browser insists the site is fine.

openssl s_client convicts the server immediately — look at how short the chain section is:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
---

One certificate. Depth 0 only. Compare that to the three-entry chain in the diagnostic section earlier — a healthy server hands over leaf plus intermediate, and the intermediate’s issuer is the root your store already trusts. With just the leaf, OpenSSL has no path to walk and reports unable to get local issuer certificate, which is why this gets misdiagnosed as a local CA problem.

You can repair the chain client-side. The leaf certificate advertises where its issuer publishes the intermediate — here is the whole repair, runnable end-to-end against example.com so you can watch it pass:

# 1. Pull the leaf certificate the server serves and ask it where its
#    issuer publishes the intermediate (the AIA extension).
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null \
  | openssl x509 -noout -text | grep -A2 "Authority Information Access"
# CA Issuers - URI:https://example.com/intermediates/r11.der   <-- the missing link

# 2. Fetch the intermediate, convert DER -> PEM, and give the file
#    a name that says exactly what it is.
curl -s https://example.com/intermediates/r11.der -o r11.der
openssl x509 -inform DER -in r11.der -out intermediate.pem

# 3. Prove the repair: verification stays on, and the request exits 0.
curl --cacert intermediate.pem https://example.com

Then point a session at it — the intermediate in your trust file completes the path, and verification stays on:

import requests

# Last resort. Scope this session to the broken host and nothing else.
s = requests.Session()
s.verify = "intermediate.pem"

resp = s.get("https://example.com/products", timeout=30)

Be clear-eyed about the trade-off. This workaround keeps your pipeline running and keeps encryption intact, but it papers over the server’s misconfiguration: every other non-browser client on the internet — curl users, mobile apps, monitoring probes — is still failing on that host. Send the operator a note about the missing intermediate. The client-side bundle is a splint, not a fix, and it expires the moment the site rotates certificates.

Verification Done Right in requests, httpx, aiohttp, and Scrapy

Once you have a merged or repaired bundle, every major scraping stack accepts it. The differences are in what each trusts by default — and that default explains several “works here, fails there” mysteries on its own:

import ssl
import requests, httpx, aiohttp

CA = "/etc/ssl/merged-ca.pem"

# requests: certifi by default
r = requests.Session()
r.verify = CA

# httpx: certifi by default
c = httpx.Client(verify=CA)

# aiohttp: the OS OpenSSL store by default, NOT certifi
ctx = ssl.create_default_context(cafile=CA)
connector = aiohttp.TCPConnector(ssl=ctx)

# Scrapy settings.py — the downloader is Twisted-based, so set the
# default OpenSSL verify path before it builds its TLS context.
import os
os.environ["SSL_CERT_FILE"] = "/etc/ssl/merged-ca.pem"

# Only matters if a middleware or pipeline shells out to requests.
os.environ["REQUESTS_CA_BUNDLE"] = "/etc/ssl/merged-ca.pem"

Stack	Default trust source	Override mechanism
requests	certifi bundle in the wheel	verify= / REQUESTS_CA_BUNDLE
httpx	certifi bundle in the wheel	verify= / SSL_CERT_FILE
aiohttp	system OpenSSL default paths	TCPConnector(ssl=ctx) with a custom context
Scrapy	Twisted over system verify paths	SSL_CERT_FILE env set early, or a custom DOWNLOADER_CLIENTCONTEXTFACTORY

That aiohttp row has bitten more than one team: the same script that failed under requests worked under aiohttp, or the reverse, purely because the two libraries consult different stores. When a discrepancy like that shows up, it’s not magic — it’s two different answers to “which roots do you trust?”

There’s a second, larger reason this table matters: certificate verification is only half of what happens in a TLS handshake, and the other half is what anti-bot systems inspect. If you want the full picture of how the handshake itself identifies your client, read TLS Fingerprinting Explained — same connection, opposite direction of scrutiny.

Why verify=False and ssl._create_unverified_context Always Backfire

You know where this goes. Deadline pressure, stack overflow answer with forty thousand upvotes, and suddenly the code says this:

# BAD. Do not ship this.
resp = requests.post(
    "https://example.com/api/login",
    json={"username": "svc-scraper", "password": os.environ["STORE_PASSWORD"]},
    verify=False,
    # Attack 1: any party on the network path now reads those credentials
    #           in plaintext and can replay the session forever.
    # Attack 2: the same attacker rewrites every page you scrape — prices,
    #           stock counts, vendor names — and your pipeline ingests it
    #           as ground truth without a single error.
)

Both attacks are concrete, not theoretical. You control neither the proxies your traffic crosses nor the networks your containers run in; an unauthenticated TLS channel means you cannot tell the real server from anyone else holding a router position. For a scraper, attack 2 is the nastier one — stolen credentials get noticed, but silently poisoned data feeds straight into dashboards and pricing decisions wearing the costume of a successful scrape. And the follow-up mistake is predictable: urllib3.disable_warnings(InsecureRequestWarning) to stop the console complaining. That’s taping over the smoke detector because the smoke is inconvenient. ssl._create_unverified_context() is the same decision wearing an async costume.

Everything covered so far reduces to one table:

Tail of the error	Actual cause	Fix that keeps verification on	Shortcut that burns you
unable to get local issuer certificate, fresh CI container	stale certifi or ca-certificates	rebuild image, upgrade the bundle	verify=False
self signed certificate in certificate chain, office network	TLS inspection proxy	merge corporate root, set REQUESTS_CA_BUNDLE	disable warnings
hostname mismatch	missing SNI, default vhost cert — or genuine interception	send correct servername/Host; treat persistent mismatch as a security event	verify=False per domain
certificate has expired / not yet valid	expired target cert, or your clock	date -u first; fix NTP or alert the site	_create_unverified_context
Chain shows depth 0 only	server omits the intermediate	fetch via AIA, scoped verify="intermediate.pem", notify the operator	global trust-everything context

Wrap-Up: The Fix Is Never “Off”

The tail of the error names the broken check. openssl s_client names the broken link in the chain. Every real fix — refresh the bundle, merge the corporate root, supply the missing intermediate, correct the clock — leaves verification intact and your data path authenticated. The one-input fix, verify=False, is the only option on the table that makes the failure invisible instead of resolved, and invisible failures in a scraping pipeline compound quietly until someone acts on the numbers.

Two habits worth adopting from here. Add a pre-flight step to your pipeline that makes one request to a known-good HTTPS host and prints the certifi version alongside it, so trust-store drift shows up as a log line instead of a 3 a.m. alert. And treat a sudden hostname mismatch across a fleet that worked yesterday as a security signal, not an inconvenience — that pattern is what active interception looks like from the client side. If you’re weighing whether to keep fighting TLS quirks in your own runtime at all, Python Web Scraping: Requests + BeautifulSoup vs Scraping API covers the trade-offs honestly, including the parts where doing it yourself still wins.