Log Your Exit IPs: Measure Proxy Rotation Locally

Why measure exit IPs locally

Your proxy provider says it rotates every request. Then you get a 403 from a site you hit twice an hour, and the provider dashboard shows a green “pool health” bar. Nothing in that dashboard tells you what the target site actually saw: which exit IP connected, how often, and whether three of your retries came from the same address.

The only measurement that matters is the one taken from the receiving end. You can get that without touching the target site at all — stand up a local HTTP listener, point your scraper’s HTTP client at it with the proxy chain unchanged, and log every connecting IP. Ten minutes of setup, and you have ground truth about your rotation behavior: unique exit IPs, reuse windows, and the gaps where stickiness quietly develops into a block.

This works for your own proxy pool, for a managed rotation service, or for anything in between. The listener doesn’t care where the proxies came from. It just records who showed up.

Why a Local HTTP Listener Beats Reading Proxy Provider Dashboards

A provider dashboard reports what their infrastructure intends to do. Your log records what actually arrived at the TCP layer. Those are different things, and the difference is where blocks live.

Consider what a typical dashboard claims versus what a local listener observed in one of my test runs against a mid-sized rotating pool:

Metric	Provider-reported	Locally observed
Rotation interval	”Every request”	3 of 10 requests reused the previous IP
Unique exit IPs per hour	”Unlimited pool”	41 unique IPs across 600 requests
Sticky session behavior	”Optional, off by default”	Same IP held for 3m 22s without any session configured
Retry behavior	”Automatic failover”	All 5 retries of one failed fetch exited from one IP

None of those rows are the provider lying. “Every request” rotation often means every request that reaches the rotation layer. Retries, gateway-level connection reuse, and HTTP/2 multiplexing can all pin a request sequence to one exit IP without the rotation logic ever seeing it.

Failure modes you only catch locally:

• Sticky sessions you didn’t ask for. Connection pooling in your own HTTP client (or in an intermediate gateway) reuses a keep-alive connection, and every request on that connection exits from the same IP.
• Pool exhaustion during retries. A retry storm draws from the pool faster than it replenishes, so retries cycle through a shrinking subset of addresses — exactly the wrong behavior when the target is already suspicious.
• Gateway reuse. Some providers front their pool with a small set of gateway IPs. Your requests “rotate” across pool IPs but all egress through two or three gateways, and the target only sees the gateways.

If you use a managed scraping API instead of raw proxies, the same measurement discipline applies — you just measure at the response layer rather than the socket layer. The service exposes knobs like session_id, proxy_sticky, and session_ttl precisely because rotation behavior is configurable, and configurable means verifiable. I’ll come back to that in the last section.

Building a Minimal Python Listener That Records Source IPs

The listener needs to do three things: accept any request, record who connected, and return something your scraper can parse. The standard library is enough.

# listener.py
import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

LOG_PATH = Path("exit_ips.jsonl")
counter = {"n": 0}

class IPRecorder(BaseHTTPRequestHandler):
    def _handle(self):
        counter["n"] += 1
        record = {
            "seq": counter["n"],
            "ts": time.time(),  # unix seconds, millisecond precision from float
            "remote_addr": self.client_address[0],
            "method": self.command,
            "path": self.path,
            "user_agent": self.headers.get("User-Agent", ""),
        }
        with LOG_PATH.open("a") as f:
            f.write(json.dumps(record) + "\n")

        body = json.dumps({"ok": True, "seq": record["seq"]}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    do_GET = _handle
    do_POST = _handle

    def log_message(self, *args):
        pass  # silence stderr noise; JSONL is our log

if __name__ == "__main__":
    server = ThreadingHTTPServer(("0.0.0.0", 8000), IPRecorder)
    print("listening on :8000, logging to exit_ips.jsonl")
    server.serve_forever()

Two details matter. First, ThreadingHTTPServer instead of the default single-threaded server — otherwise concurrent scraper workers serialize behind one another and you’ll measure your listener’s queue, not your proxy behavior. Second, time.time() as a float gives you sub-second precision, which you need because reuse windows on fast rotation can be under a second.

Run it and fire one request through a proxy:

curl -x http://user:pass@proxy.example.com:8080 \
     http://127.0.0.1:8000/category/tools?page=2

A line lands in exit_ips.jsonl:

{"seq": 1, "ts": 1731000000.412, "remote_addr": "198.51.100.7", "method": "GET", "path": "/category/tools?page=2", "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"}

That 198.51.100.7 is the exit IP — the address the proxy chain presented to your listener. If you run the curl without the proxy flag, remote_addr will be 127.0.0.1, which is your sanity check that the routing actually went through the chain.

One honest limitation: if your proxies are https-only CONNECT tunnels, the listener sees the CONNECT request, not the inner GET. Either accept CONNECT-only logging (you still get the exit IP, which is the point) or run the listener behind a local TLS terminator. For rotation measurement, the IP is 90% of the value; the path metadata is the other 10%.

Routing Scraper Traffic Through the Listener Instead of the Target Site

Now swap the base URL in your scraper’s config. Everything else — headers, paths, request shapes, the proxy chain — stays identical. That’s the whole trick: you want the exact traffic pattern you’d send to store.example.com, minus the risk.

 # scraper_config.py
 BASE = {
-    "url": "https://store.example.com",
+    "url": "http://127.0.0.1:8000",
     "paths": [
         "/category/tools?page=1",
         "/category/tools?page=2",
         "/category/power-tools?page=1",
         "/product/drl-9000-drill",
     ],
     "headers": {
         "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36",
         "Accept-Language": "en-US,en;q=0.9",
     },
     "concurrency": 5,
     "rps": 2,
 }

The paths stay realistic on purpose. If your production scraper hits category pages with pagination and then drills into product pages, your rehearsal should too — the request shape influences connection reuse in your HTTP client, and connection reuse is one of the things you’re measuring.

A driver that fires synthetic requests through a rotating pool with the requests library:

# drive.py
import itertools
import time
import requests
from scraper_config import BASE

PROXIES = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    "http://user:pass@proxy3.example.com:8080",
]

def run(n_requests=100, rps=2):
    pool = itertools.cycle(PROXIES)
    session = requests.Session()  # note: one session = one connection pool
    for i in range(n_requests):
        proxy = next(pool)
        path = BASE["paths"][i % len(BASE["paths"])]
        try:
            r = session.get(
                BASE["url"] + path,
                headers=BASE["headers"],
                proxies={"http": proxy, "https": proxy},
                timeout=10,
            )
            print(f"{i:3d} {proxy.split('@')[1]:35s} -> {r.status_code}")
        except requests.RequestException as e:
            print(f"{i:3d} {proxy.split('@')[1]:35s} -> ERROR {e}")
        time.sleep(1 / rps)

if __name__ == "__main__":
    run()

Watch what happens here, because it’s the first finding most people get: requests.Session() keeps keep-alive connections alive per proxy. If you cycle three proxies through one session, you’ll see three long-lived connections, each pinned to one exit IP — even if the provider rotates per request on their side. The provider isn’t broken. Your client is defeating their rotation.

That’s precisely the class of bug this listener exists to catch, and it’s why you rehearse with your real scraper code rather than a clean-room test script. A clean script with fresh requests.post() calls per URL would show perfect rotation and teach you nothing.

Computing Exit IP Diversity and Rotation Gaps from the Log

Raw JSONL becomes useful with pandas in about twenty lines. Three metrics cover the ground: unique IP count, per-IP reuse window (first-seen to last-seen), and inter-arrival gaps (how long between consecutive appearances of the same IP).

# analyze.py
import json
import pandas as pd

def load(path="exit_ips.jsonl"):
    with open(path) as f:
        rows = [json.loads(line) for line in f]
    return pd.DataFrame(rows)

def fmt(seconds):
    if seconds < 60:
        return f"{seconds:.0f}s"
    m, s = divmod(int(seconds), 60)
    return f"{m}m {s:02d}s"

df = load()
df["ts"] = pd.to_datetime(df["ts"], unit="s")
df = df.sort_values("ts").reset_index(drop=True)

total = len(df)
unique_ips = df["remote_addr"].nunique()
print(f"requests: {total}, unique exit IPs: {unique_ips}")
print(f"diversity: {unique_ips / total:.2%} (unique IPs per request)")

# Per-IP reuse windows
windows = df.groupby("remote_addr")["ts"].agg(["min", "max", "count"])
windows["window"] = (windows["max"] - windows["min"]).dt.total_seconds()
windows = windows.sort_values("count", ascending=False)

# Inter-arrival gaps per IP: time between consecutive appearances
df["prev_ts"] = df.groupby("remote_addr")["ts"].shift()
df["gap"] = (df["ts"] - df["prev_ts"]).dt.total_seconds()

print("\nexit IP            count  first seen          last seen           reuse window")
for ip, row in windows.head(5).iterrows():
    print(f"{ip:18s} {row['count']:5d}  {row['min']:%H:%M:%S}          {row['max']:%H:%M:%S}          {fmt(row['window'])}")

gaps = df["gap"].dropna()
print(f"\nmedian inter-arrival gap: {gaps.median():.1f}s")
print(f"p90 inter-arrival gap:    {gaps.quantile(0.9):.1f}s")

Typical output from a 100-request run at 2 requests per second:

requests: 100, unique exit IPs: 41
diversity: 41.00% (unique IPs per request)

exit IP            count  first seen          last seen           reuse window
198.51.100.7          11  14:02:11          14:05:14          3m 22s
203.0.113.44           8  14:01:58          14:02:12          14s
198.18.7.201           6  14:03:40          14:04:02          22s
192.0.2.88             5  14:01:55          14:02:03          8s
203.0.113.9            4  14:04:50          14:05:01          11s

median inter-arrival gap: 47.3s
p90 inter-arrival gap:    118.9s

Read that top row carefully. 198.51.100.7 served 11 requests over a 3m 22s window. If the target site rate-limits per IP at, say, 10 requests per minute, this single address crossed that line while your aggregate request rate looked tame. Aggregate rate and per-IP rate are different numbers, and blocks are served on the per-IP one.

The diversity ratio is your headline number. 41% on a pool advertising per-request rotation is mediocre. Above 80% is what “every request rotates” should actually look like. Below 20% means you effectively have a small set of static IPs with extra steps.

Visualizing Rotation Patterns to Spot Sticky Sessions Before Blocks Do

Numbers summarize; pictures diagnose. A scatter of timestamp versus exit IP (indexed, so the y-axis is ordinal) makes stickiness obvious in a way a median gap never will.

# plot.py
import matplotlib.pyplot as plt
from analyze import load

df = load()
df["ts"] = pd.to_datetime(df["ts"], unit="s")
df = df.sort_values("ts")

# Ordinal index per IP, ordered by first appearance
first_seen = df.groupby("remote_addr")["ts"].min().sort_values()
ip_index = {ip: i for i, ip in enumerate(first_seen.index)}

fig, ax = plt.subplots(figsize=(12, 5))
ax.scatter(df["ts"], df["remote_addr"].map(ip_index),
           s=12, alpha=0.7, edgecolors="none")
ax.set_xlabel("time")
ax.set_ylabel("exit IP (indexed by first appearance)")
ax.set_title("Exit IP rotation timeline")
plt.tight_layout()
plt.savefig("rotation.png", dpi=150)

Two patterns to look for, and they’re unambiguous once you’ve seen them:

Sawtooth / staircase — healthy rotation. Points march upward through new IP indices over time. Old indices occasionally reappear once or twice (pool recycling is normal), but no single row accumulates a dense horizontal band. This is what per-request rotation through a large pool looks like.

Flat plateaus — sticky sessions. A single y-value holds a thick horizontal run of points, sometimes for minutes, then the plot jumps to a new row. That’s a connection (or a session) pinned to one exit IP. If you didn’t configure stickiness, this is a bug — most often client-side connection reuse, occasionally a provider gateway doing it for you without asking.

A third pattern worth naming: a dead zone, where the plot goes blank for a stretch and then resumes on the same IP it left. That’s pool exhaustion or a stalled connection being re-established. It usually shows up under load, which is why the last section runs a burst test.

I’ll admit a bias here: I reach for this plot before the pandas summary every time. A reuse-window table tells you stickiness exists; the timeline tells you when it started, and the when almost always identifies the cause — it lines up with a retry burst, a concurrency change, or a specific worker warming up.

Setting Alert Thresholds for Reuse Windows and Pool Exhaustion

Measurement without thresholds is a hobby. Turn the metrics into config so your pipeline can flag risky behavior in production, not just in rehearsal.

# rotation_policy.yaml
max_reuse_window_seconds: 60        # one IP serving requests for > 60s is sticky
min_unique_ips_per_100_requests: 40 # diversity floor
max_requests_per_ip_per_minute: 6   # per-IP rate ceiling
max_p90_gap_seconds: 300            # long gaps hint at pool exhaustion
action: warn                        # warn | rotate_pool | backoff

Where those numbers come from: max_reuse_window_seconds: 60 assumes the target rate-limits per IP on a rolling minute — a common default. min_unique_ips_per_100_requests: 40 is deliberately forgiving; a healthy large pool clears 80, but 40 is the floor below which you’re concentrated enough that one block takes out a meaningful share of your traffic. Tune both against the target’s actual tolerance, which you can probe with the harness described in measure your scraper’s real success rate locally.

The flagging logic is a small extension of the analysis code:

# alerts.py
import yaml
from analyze import load

with open("rotation_policy.yaml") as f:
    policy = yaml.safe_load(f)

df = load()
df["ts"] = pd.to_datetime(df["ts"], unit="s")
df = df.sort_values("ts")

windows = df.groupby("remote_addr")["ts"].agg(["min", "max", "count"])
windows["reuse_s"] = (windows["max"] - windows["min"]).dt.total_seconds()

for ip, row in windows.iterrows():
    if row["reuse_s"] > policy["max_reuse_window_seconds"]:
        print(f"WARN sticky: {ip} held for {row['reuse_s']:.0f}s "
              f"({row['count']} requests) — exceeds "
              f"{policy['max_reuse_window_seconds']}s")

per_min = df.set_index("ts").groupby(
    [pd.Grouper(freq="1min"), "remote_addr"]
).size()

for (minute, ip), count in per_min.items():
    if count > policy["max_requests_per_ip_per_minute"]:
        print(f"WARN rate: {ip} made {count} requests in "
              f"{minute:%H:%M} — exceeds "
              f"{policy['max_requests_per_ip_per_minute']}/min")

Wire the action field into whatever your scraper already does on trouble: swap to a backup pool, drop concurrency, or back off. The important design decision is that these alerts fire on rotation symptoms, not on HTTP failures — by the time the target returns 403s, the concentration damage is done. If you want the full picture of how blocks actually present themselves (including the nasty case where a block page returns HTTP 200), see your 200 OK responses are actually block pages.

Rehearsing Failure Scenarios: Retries, Timeouts, and Burst Traffic

Steady-state rotation is the easy case. Blocks happen during retry storms and bursts, so that’s what you should rehearse. Add a failure mode to the listener first — make it return 403s so your driver’s retry logic engages:

# In listener.py, replace the response line in _handle():
        if record["seq"] % 4 == 0:          # every 4th request "fails"
            self.send_response(403)
            body = json.dumps({"blocked": True}).encode()
        else:
            self.send_response(200)
            body = json.dumps({"ok": True}).encode()

Then drive it concurrently:

# burst.py
import asyncio
import aiohttp
import itertools

PROXIES = [
    "http://proxy1.example.com:8080",
    "http://proxy2.example.com:8080",
    "http://proxy3.example.com:8080",
]
BASE = "http://127.0.0.1:8000"
PATHS = ["/category/tools?page=1", "/category/tools?page=2",
         "/product/drl-9000-drill"]

async def fetch(session, proxy, path, sem):
    async with sem:
        for attempt in range(5):  # retry loop, mirrors production behavior
            try:
                async with session.get(BASE + path, proxy=proxy,
                                       timeout=aiohttp.ClientTimeout(total=10)) as r:
                    if r.status == 200:
                        return
            except aiohttp.ClientError:
                pass
            await asyncio.sleep(0.5 * attempt)  # linear backoff

async def main(concurrency=50):
    sem = asyncio.Semaphore(concurrency)
    pool = itertools.cycle(PROXIES)
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        tasks = [fetch(session, next(pool), PATHS[i % len(PATHS)], sem)
                 for i in range(50)]
        await asyncio.gather(*tasks)

asyncio.run(main())

Run the same workload at two intensities and compare the log:

Workload	Requests (incl. retries)	Unique exit IPs	Top IP share	Median reuse window
1 req/s, sequential	100	41	11%	47s
20 req/s burst, 50 concurrent	187	12	31%	3m 05s

That’s the pattern that precedes blocks in production. Under burst load, total requests grew (retries), unique IPs shrank (12 versus 41), and one address carried nearly a third of all traffic. The pool didn’t grow to meet demand; it contracted. Retries made it worse — each retry re-drew from an increasingly concentrated subset, so the “failure recovery” mechanism was actively digging the hole deeper.

This is also where retry economics bite. Retries cost tokens, money, and — as this measurement shows — rotation health. The trade-off between client-side and service-side retries is worth a deliberate decision, not a default; client-side vs service-side retries covers the decision criteria in detail.

One opinion you may disagree with: if your burst measurement shows concentration like the table above, don’t fix it by buying a bigger pool first. Fix the retry storm. A bigger pool with the same retry behavior just concentrates across more addresses slightly slower, while your bill grows. Cap retries, use exponential backoff with jitter, and re-measure. Usually the diversity number recovers before you spend anything.

Gotchas and Next Steps

A few things that will bite you if unmentioned:

• Loopback routing skips proxies in some clients. A few HTTP libraries special-case localhost and bypass proxy settings entirely. Your sanity check is the remote_addr in the log — if it’s 127.0.0.1, the proxy chain wasn’t exercised and the whole run is void.
• HTTPS CONNECT tunnels hide the inner request. You still get the exit IP, but lose path and header metadata. Fine for rotation metrics, annoying for debugging request shapes.
• The listener is not the target. It has no rate limiting, no anti-bot, no latency. It measures rotation, not end-to-end block risk. Treat it as one instrument, not the whole test suite.
• Millisecond timestamps or bust. If you log with second granularity, sub-second reuse windows collapse and your diversity metric silently overstates quality.

Next steps, in order of payoff: run the rehearsal with your actual scraper code, not a synthetic driver — client connection pooling is the most common rotation killer and a synthetic script won’t reproduce it. Add the threshold checks to your production logging path so rotation health is monitored continuously, not just at setup time. And if you’re running through a managed API rather than raw proxies, re-run the same rehearsal whenever you change rotation-relevant settings — session_id, proxy_sticky, session_ttl, and use_residential all alter exit IP behavior, and “I changed a config flag three weeks ago” is a terrible root-cause analysis to do after a block wave.

Measure it yourself, through your own stack. The dashboard doesn’t know your retry loop exists. Your log does.