Audit What Your Scraper Sends With a Local Echo Server

Why Your Scraper’s Outgoing Requests Matter More Than Its Proxies

Most scraping debugging starts at the wrong end. You get blocked, so you rotate proxies. Still blocked, so you buy residential proxies. Still blocked, so you blame the proxy vendor. Meanwhile the actual problem is that every request you send announces “I am a Python script” in about six different headers, and no IP rotation fixes that.

Anti-bot systems score requests across multiple signals. IP reputation is one input, but header order, header values, cookie behavior, and the TLS handshake often carry more weight. A residential IP sending a request with python-requests/2.x in the User-Agent and no Accept-Language is more suspicious than a datacenter IP sending a request that looks byte-for-byte like Chrome. The IP is a proxy away from fixed. The fingerprint travels with your code.

Here’s the side-by-side. This is what a real browser sends to https://example.com versus what requests.get('https://example.com') sends by default:

Header	Chrome (real browser)	Python requests (default)
User-Agent	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ... Chrome/131.0.0.0 Safari/537.36	python-requests/2.32.3
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8	*/*
Accept-Language	en-US,en;q=0.9	absent
Accept-Encoding	gzip, deflate, br, zstd	gzip, deflate (no br, no zstd)
sec-ch-ua	"Chromium";v="131", "Not_A Brand";v="24"	absent
sec-ch-ua-mobile	?0	absent
sec-ch-ua-platform	"Windows"	absent
sec-fetch-dest	document	absent
sec-fetch-mode	navigate	absent
sec-fetch-site	none	absent
upgrade-insecure-requests	1	absent

A raw capture of the requests call makes it obvious:

GET / HTTP/1.1
Host: example.com            <-- fine
Accept-Encoding: gzip, deflate  <-- no br/zstd; a browser never sends this exact set
Accept: */*                  <-- no browser navigates with a bare */*
User-Agent: python-requests/2.32.3   <-- the confession
Connection: keep-alive

Four headers total. A real browser sends fifteen-plus, in a specific order, with client hints. Any WAF on the receiving end can score this in microseconds. If you want the full picture of how these signals get combined with the TLS handshake, read TLS Fingerprinting Explained — the short version is that the JA3/JA4 hash of your handshake and your header set need to tell the same story, or you fail.

The fix is not guessing. It’s looking. You need to see exactly what your scraper transmits before an anti-bot system grades it for you.

Building a Minimal Echo Server That Reflects Every Byte Back

The idea is simple: run a local HTTP server that does nothing except reflect the incoming request back as JSON — every header, every cookie, the body, the client port. Point your scraper at it. Read what comes back. Thirty lines of Flask does it:

# echo_server.py
import json
from flask import Flask, request

app = Flask(__name__)

@app.route("/", defaults={"path": ""})
@app.route("/<path:path>")
@app.route("/<path:path>", methods=["POST", "PUT", "DELETE", "PATCH"])
def echo(path):
    payload = {
        "method": request.method,
        "path": "/" + path,
        "args": request.args.to_dict(),
        "headers": dict(request.headers),
        "cookies": request.cookies.to_dict(),
        "body": request.get_data(as_text=True)[:2000],
        "client": {"ip": request.remote_addr, "port": request.remote_port},
    }
    print(json.dumps(payload, indent=2))       # console log
    with open("echo_log.jsonl", "a") as f:     # durable log
        f.write(json.dumps(payload) + "\n")
    return json.dumps(payload, indent=2), 200, {"Content-Type": "application/json"}

if __name__ == "__main__":
    app.run(host="127.0.0.1", port=8000)

Run it with python echo_server.py. Every request — any method, any path — gets logged to echo_log.jsonl and returned verbatim to the caller, so your scraper sees a “successful” response and behaves normally.

A GET from a scraper hitting http://localhost:8000/product/123 comes back like this, with each field telling you something:

{
  "method": "GET",
  "path": "/product/123",
  "args": {},
  "headers": {
    "Host": "localhost:8000",              // tells you where it actually went
    "User-Agent": "python-requests/2.32.3", // the leak
    "Accept-Encoding": "gzip, deflate",     // no br — mismatch with a Chrome UA claim
    "Accept": "*/*",                        // no browser sends this
    "Connection": "keep-alive"
  },
  "cookies": {},                            // should be empty on a cold run
  "body": "",
  "client": {"ip": "127.0.0.1", "port": 53114}
}

One caveat worth stating up front: Flask gives you the headers as parsed by the WSGI server, not the raw bytes. Header order is lost, and some servers normalize header casing. If you need byte-exact ordering — and eventually you will, because anti-bot systems check it — swap the Flask front for a raw socket or asyncio.start_server implementation that reads the request bytes before parsing. For a first-pass audit, parsed headers catch 90% of the leaks.

Pointing Your Scraper at the Echo Server Without Changing Production Code

An audit that requires editing your scraper’s source is an audit you’ll skip under deadline pressure. The redirect has to be configuration, not code.

If your scraper reads its target from a constant or config, the cheapest path is an environment variable override:

# config.py
import os

BASE_URL = os.environ.get("SCRAPE_TARGET", "https://store.example.com")

SCRAPE_TARGET=http://localhost:8000 python scraper.py

Zero code changes. The scraper fetches http://localhost:8000/product/123 believing it’s talking to the store, and the echo log fills up with its true outgoing behavior.

The env-var trick breaks the moment your code touches TLS-specific behavior, hardcodes https://, or validates the response against the real site’s markup. For those cases, use a hosts-file override instead:

# /etc/hosts  (Linux/macOS)  or  C:\Windows\System32\drivers\etc\hosts
127.0.0.1   store.example.com

Now https://store.example.com/product/123 resolves to your machine. The scraper keeps its real URL, real Host header, real scheme — everything except the destination. The catch: your echo server now needs to present a TLS certificate that the scraper’s HTTP client will accept. Generate a self-signed cert for store.example.com, run the echo server behind it, and either add the cert to your local trust store or disable verification in the test run only:

openssl req -x509 -newkey rsa:2048 -nodes -days 30 \
  -keyout key.pem -out cert.pem -subj "/CN=store.example.com"

app.run(host="127.0.0.1", port=443,
        ssl_context=("cert.pem", "key.pem"))

The hosts approach is more work, but it’s the only way to audit code paths that branch on TLS or that pin the scheme. I use the env var for quick checks and the hosts file when I’m chasing something subtle.

One thing the local echo server cannot capture: the TLS handshake itself. The client hello — cipher suites, extensions, their order — happens below HTTP. Your echo log shows headers only. That’s fine; the handshake is a separate problem with separate tooling, and mixing the two audits muddies both.

Capturing and Diffing Requests Across Scraping Libraries

Here’s where the echo server starts earning its keep. Run the identical logical request through three common libraries and diff the results. Same target path, same “scrape the product page” intent:

# requests
import requests
requests.get("http://localhost:8000/product/123")

# httpx
import httpx
with httpx.Client() as client:
    client.get("http://localhost:8000/product/123")

# Playwright
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("http://localhost:8000/product/123")
    browser.close()

The echo log for each, condensed to the interesting fields:

Header / behavior	requests	httpx	Playwright (Chromium)
User-Agent	python-requests/2.x	python-httpx/0.2x	Full Chrome UA
Accept	*/*	*/*	text/html,application/xhtml+xml,...
Accept-Language	absent	absent	en-US (settable via context locale)
Accept-Encoding	gzip, deflate	gzip, deflate (HTTP/1.1) / gzip, br (H2)	gzip, deflate, br
sec-ch-ua family	absent	absent	present
sec-fetch-* family	absent	absent	present
upgrade-insecure-requests	absent	absent	1
HTTP version	1.1	1.1 or 2.0 depending on config	2.0
Header order	library-specific	library-specific	Chromium’s order

The pattern that jumps out: the two plain HTTP libraries are nearly identical in what they omit, while the headless browser gets the header set right by construction. That’s the honest trade-off. Playwright’s defaults are close to a real browser’s — but its handshake and runtime still carry headless tells, which is a whole separate audit covered in Headless Browser Detection. Plain clients are fast and cheap but need you to reconstruct the header set by hand, and they’ll never get the TLS layer right on their own.

A detail people miss: Accept-Encoding changes with protocol version in httpx. If your production traffic runs over HTTP/2 but your echo test ran over HTTP/1.1 (because you pointed at a plain Flask server), you audited a fingerprint you don’t actually ship. Match the protocol in the test to the protocol in production, or the diff is fiction.

Detecting Cookie and Session Leaks in the Echo Logs

Headers are the visible leak. Cookies are the quiet one. A session cookie from yesterday’s test run riding along on today’s request can link your “fresh” identity to a previously flagged one — and you’d never notice without looking.

Here’s a real-shaped excerpt from echo_log.jsonl. Two runs, five minutes apart, supposedly independent:

{"ts": "run-1", "cookies": {}}
{"ts": "run-2", "cookies": {
   "session_id": "a9f3c2e1...",      // <-- carried over from run 1
   "_ab_test": "variant_b"
}}

That session_id should not exist in run 2. Where did it come from? A shared cookie jar in a module-level client, a persisted session file, or a library defaulting to cookie persistence across what you assumed were isolated requests. On the target site’s side, that cookie ties run 2 to run 1’s identity — including run 1’s rate-limit strikes. This is how scrapers get shadow-banned while looking perfectly clean from their own logs.

The fix in httpx is explicit cookie scoping:

import httpx

# Fresh jar per logical session — no cross-contamination
def make_client():
    return httpx.Client(
        cookies=httpx.Cookies(),
        headers={...},
    )

# Or clear selectively between request groups
client = httpx.Client()
client.get("http://localhost:8000/product/123")
client.cookies.clear()   # drop everything before the next identity
client.get("http://localhost:8000/product/456")

If you’re using sticky sessions deliberately — warmup sequences, multi-step flows where you want the same identity — that’s a legitimate pattern, and Sticky Exit IPs for Warmup and Follow-Up Scrapes covers how to do it on purpose. The failure mode isn’t cookie persistence existing; it’s persistence you didn’t ask for. The echo log is how you tell the two apart.

Also check for cookies you never set. Headless browsers arrive with defaults, and some middleware injects tracking cookies. If the echo shows a cookie with no corresponding Set-Cookie in a prior logged response, something in your own stack created it. Find that before the target site does.

Fixing the Leaks: Rewriting Headers and TLS Fingerprints

Once the echo log shows the gaps, remediation is mechanical. Before:

# Default httpx — announces itself in two headers
import httpx

client = httpx.Client()
r = client.get("https://store.example.com/product/123")

After:

import httpx

BROWSER_HEADERS = {
    "User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                   "AppleWebKit/537.36 (KHTML, like Gecko) "
                   "Chrome/131.0.0.0 Safari/537.36"),
    "Accept": ("text/html,application/xhtml+xml,application/xml;q=0.9,"
               "image/avif,image/webp,*/*;q=0.8"),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br, zstd",
    "sec-ch-ua": '"Chromium";v="131", "Not_A Brand";v="24"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "navigate",
    "sec-fetch-site": "none",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
}

client = httpx.Client(headers=BROWSER_HEADERS, http2=True)
r = client.get("https://store.example.com/product/123")

Re-run against the echo server and the diff confirms the header set now matches a browser’s. But headers are only half the story. The TLS client hello still says “Python’s ssl module,” and any system that checks JA3/JA4 will see a Chrome User-Agent riding a Python handshake — a mismatch that’s arguably worse than honest defaults, because it looks like deliberate spoofing.

To align the handshake, use a library that impersonates a browser’s TLS stack. curl_cffi is the one I reach for:

from curl_cffi import requests

# Impersonates Chrome's TLS handshake AND HTTP/2 settings
r = requests.get(
    "https://example.com",
    impersonate="chrome",
)
print(r.status_code)

That single impersonate argument fixes the cipher suite list, extension order, ALPN behavior, and HTTP/2 frame settings to match a real Chrome client. Combined with the header set above, your request is consistent from the SYN packet to the last header byte.

The alternative is not doing any of this yourself. A scraping API handles the fingerprint layer for you — for example, one call to the sync scrape endpoint with a TLS profile:

curl -X POST https://api.finedata.ai/api/v1/scrape \
  -H "Authorization: Bearer fd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/product/123",
    "formats": ["markdown"],
    "tls_profile": "chrome136",
    "use_antibot": true
  }'

The tls_profile parameter pins the handshake to a specific browser fingerprint, and use_antibot is on by default. You still want the echo-server audit in that world — your headers, cookies, and callback configuration remain your responsibility — but the TLS and HTTP/2 layers stop being your problem. My honest position: hand-rolling with curl_cffi is fine for one target you control; once you’re juggling multiple sites with different anti-bot stacks, the maintenance tax of staying ahead of fingerprint drift starts exceeding the API bill. Web Scraping API vs DIY: Total Cost of Ownership runs those numbers properly.

Automating the Audit: Fail Your Tests When a Fingerprint Regresses

Manual echo inspection catches leaks once. It won’t catch the regression six weeks later when a library upgrade silently changes your User-Agent or a dependency starts injecting a header. Turn the audit into a test.

The pattern: start the echo server, fire one scraper request, assert the outgoing header set matches a stored baseline.

# test_fingerprint.py
import json, subprocess, time
import httpx
import pytest

BASELINE_FILE = "tests/baseline_headers.json"

@pytest.fixture(scope="module")
def echo_server():
    proc = subprocess.Popen(["python", "echo_server.py"])
    time.sleep(1.5)
    yield "http://localhost:8000"
    proc.terminate()

def test_outgoing_headers_match_baseline(echo_server):
    # One representative scraper request
    with httpx.Client(headers=scraper_headers()) as client:
        client.get(f"{echo_server}/product/123")

    # Read back what the server actually received
    echoed = json.loads(open("echo_log.jsonl").readlines()[-1])
    actual = {k.lower(): v for k, v in echoed["headers"].items()
              if not k.lower().startswith(("host", "content-length"))}

    baseline = json.load(open(BASELINE_FILE))

    missing = set(baseline) - set(actual)
    extra = set(actual) - set(baseline)
    assert not missing, f"Headers dropped: {sorted(missing)}"
    assert not extra, f"Headers added: {sorted(extra)}"
    for k in baseline:
        if k in actual:
            assert actual[k] == baseline[k], f"Value changed: {k}"

Generate the baseline once, review it by hand, commit it. Then a routine dependency bump produces this in CI:

FAILED tests/test_fingerprint.py::test_outgoing_headers_match_baseline
AssertionError: Value changed: user-agent
  baseline: Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/131.0.0.0 ...
  actual:   python-httpx/0.27.0

That’s a library upgrade reverting your header override — the exact class of silent regression that costs days of “why did our success rate drop” investigation. Caught here, it’s a five-minute fix. The same harness catches new cookies appearing in the jar and headers you removed by accident.

Two things to exclude from the baseline: Host (varies with the redirect target) and Content-Length (varies with the body). Everything else should be deterministic. If it isn’t, that nondeterminism is itself a finding — a header that changes between runs is a signal some anti-bot systems specifically look for.

Wrap-Up and Gotchas

The echo server is the cheapest diagnostic tool in scraping: an hour to build, zero production impact, and it shows you the exact bytes your scraper transmits instead of what you assume it transmits. Headers, cookies, and protocol behavior — all visible, all diffable, all testable in CI.

Gotchas to keep in mind:

• Flask normalizes headers. You lose order and original casing. Fine for a first pass; move to a raw-socket echo when you need byte-exact captures.
• The echo server can’t see TLS. Client hello fingerprints need separate tooling — capture with a TLS proxy or check what your library’s impersonation mode actually emits.
• Match protocols in test and prod. An HTTP/1.1 echo test of an HTTP/2 production client audits the wrong fingerprint.
• Baseline the boring headers too. Accept-Encoding mismatches have sunk more scrapers than User-Agent strings, because everyone remembers to fix the UA and nobody remembers the encoding list.

Next steps: wire the pytest baseline into your CI pipeline this week, before the next dependency bump. Then extend the harness to run one request through every client your stack uses — the diff table from earlier in this post is the format I’d use for the report. If you’re routing through a scraping API, the same technique applies to your callback and webhook configuration: point the callback_url at the echo server in a test job and verify what your pipeline actually receives. Measure first. Fix second. Always in that order.