Your Scraped Links Don't Resolve: Fix Relative URLs

Why ‘/products/42’ Breaks Your Crawl: The Anatomy of a Relative URL

You scrape a listing page, extract 200 hrefs, and push them into your crawl queue. Six hours later the worker logs are full of GET /products/42 — invalid URL. Your scheduler treats the same product as three distinct pages because it saw /products/42 once and https://example.com/products/42 twice. The dedupe table is now lying to you, and nobody noticed until the downstream dataset shipped.

The root cause is simple: browsers resolve relative URLs against the page they appear on. Your scraper doesn’t. It stores the raw href attribute and moves on, assuming someone downstream will fix it. Nobody does.

Here’s how the same relative href resolves differently depending on where it was found:

href = "/products/42"

Found on page                          Resolves to
─────────────────────────────────────  ───────────────────────────────────────
https://example.com/listing?page=2     https://example.com/products/42
https://store.example.com/listing      https://store.example.com/products/42
https://api.example.com/docs/index     https://api.example.com/products/42  ← wrong site entirely

A leading slash means “root of this host.” That’s it. The href carries no host information at all, so the host comes from wherever the link was extracted. Store the href raw and you’ve stored half a URL.

The damage shows up in three specific places:

Failure point	Stored relative URL	Stored absolute URL
Dedupe keys	/products/42 from two hosts collide or fragment — same key for different pages, different keys for same page	Host is explicit; keys are stable
Fetch after domain change	/products/42 is unresolvable without re-deriving the base	https://example.com/products/42 still fetches, or fails cleanly
Cross-domain link graph	You cannot build one — edges have no target host	Graph edges are fully qualified

The third one hurts the most if you ever want to compute domain-level stats, PageRank-style metrics, or even just “which domains does this site link to most.” A link graph built from relative hrefs is a graph with unlabeled nodes.

The Base URL Is the Missing Input: What RFC 3986 Resolution Actually Requires

URL resolution is a pure function of two inputs: the relative reference and a base URI. RFC 3986 section 5 defines the algorithm, and it needs the base’s scheme, authority, and path to produce anything. If you don’t supply a base, you cannot resolve — full stop.

Python’s standard library implements this correctly:

from urllib.parse import urljoin

# Leading slash: absolute path, resolved against the base's authority
urljoin("https://example.com/listing?page=2", "/products/42")
# 'https://example.com/products/42'

# No leading slash: resolved against the base's *directory*
urljoin("https://example.com/listing?page=2", "products/42")
# 'https://example.com/products/42'   ← lucky: /listing has no directory part

urljoin("https://example.com/catalog/listing?page=2", "products/42")
# 'https://example.com/catalog/products/42'   ← NOT what most people expect

That second pair is the trap. A relative href without a leading slash resolves against the base URL’s path directory, not the root. If your listing pages live at /catalog/listing, every products/42 href on them points to /catalog/products/42, which is a 404 on most sites. This is correct RFC behavior. It just surprises people who assumed “relative means prepend the domain.”

Now the harder question: which base? You have three candidates, and they produce different answers:

from urllib.parse import urljoin

href = "/products/42"

# Candidate 1: the URL you requested
requested = "https://example.com/listing?page=2"
urljoin(requested, href)
# 'https://example.com/products/42'

# Candidate 2: the URL you actually landed on after redirects
final = "https://store.example.com/listing?page=2"
urljoin(final, href)
# 'https://store.example.com/products/42'

# Candidate 3: the <base> tag in the HTML head
# <base href="https://cdn.example.com/catalog/">
base_tag = "https://cdn.example.com/catalog/"
urljoin(base_tag, href)
# 'https://cdn.example.com/products/42'

The right answer, in nearly all cases, is candidate 2: the final URL after redirects. The HTML was served from that URL. Every relative reference in it was authored against that address. Using the original request URL means resolving links against a host that may not even serve the page anymore.

The <base> tag technically wins if present — the browser uses it — but in practice it’s rare and frequently wrong (leftover from a CMS migration). My rule: honor <base> if present, otherwise use the final response URL, and log a warning when a <base> tag exists because it usually means the site’s links won’t behave the way you assumed.

One-Line Fix in Python: urljoin() and Where People Get It Wrong

The standard library does the heavy lifting. What you need is a thin wrapper that handles the cases where urljoin alone is wrong or pointless:

from urllib.parse import urljoin, urlsplit

def resolve_link(href: str, base: str) -> str | None:
    """Resolve a scraped href against the page's final URL.
    Returns None for hrefs that should not be stored at all."""
    if not href or not href.strip():
        return None                      # empty or whitespace-only

    href = href.strip()

    if href.startswith("#"):
        return None                      # in-page anchor, not a document

    if urlsplit(href).scheme in ("http", "https") and href.startswith("http"):
        return href                      # already absolute — pass through

    if ":" in href.split("/")[0] and not href.startswith("//"):
        return None                      # mailto:, tel:, javascript: — skip

    return urljoin(base, href)

The guards matter more than the urljoin call. Empty hrefs are common on real pages — placeholder <a> tags in templates. mailto: links will happily pass through urljoin unchanged and then sit in your crawl queue as garbage. And anchor-only links like #reviews resolve to the current page, which means you’d queue a duplicate fetch of a page you already have.

Two mistakes I see constantly:

Mistake 1: applying resolution inconsistently. Someone writes the resolver, applies it in one ingestion path, and forgets the nightly batch job that also inserts links. Now the table has a mix of relative and absolute URLs and the dedupe problem is worse than before, because /products/42 and https://example.com/products/42 coexist. Resolution must happen at exactly one place: the boundary between scrape and storage.

Mistake 2: the wrong base after redirects. Here’s the counter-example, spelled out:

# You requested example.com but got redirected to store.example.com
requested_url = "https://example.com/listing"
final_url     = "https://store.example.com/listing"

href = "/products/42"

# BUG: using the original request URL as base
urljoin(requested_url, href)
# 'https://example.com/products/42'   ← wrong host, page moved away from here

# CORRECT: using the final response URL
urljoin(final_url, href)
# 'https://store.example.com/products/42'

If you’re using a scraping API, make sure you capture the final URL from the response rather than assuming it equals what you sent. With the FineData API, a POST to /api/v1/scrape returns the fetched page content — request it with formats: ["rawHtml"] and treat the response’s final URL as your base, not your input. The same discipline applies to any fetch layer: requests gives you response.url, Playwright gives you page.url. Use those.

If you want a refresher on how redirects silently change your assumptions, I covered the related failure mode of block pages that return 200 OK in a previous post — same class of bug, where what you asked for and what you got diverge.

Handling the Edge Cases: Protocol-Relative, Anchor, and Malformed Hrefs

Real pages contain hrefs that naive resolution mangles. Here is the full decision table:

href value	Resolution rule	Resulting absolute URL
//store.example.com/x	Protocol-relative: inherit scheme from base	https://store.example.com/x
#section	Skip — in-page anchor	(not stored)
“ (empty)	Skip — placeholder	(not stored)
products/42	Resolve against base’s path directory	https://example.com/catalog/products/42
/products/42	Resolve against base authority	https://example.com/products/42
https://example.com/a	Already absolute — pass through	https://example.com/a
mailto:support@example.com	Skip — not an HTTP resource	(not stored)

Protocol-relative URLs (//store.example.com/x) deserve a note. They were popular during the http/https transition era and still appear on older sites. urljoin handles them correctly by inheriting the base’s scheme, so they resolve fine — but be aware that if your base is http:// for some reason, you’ll generate an insecure URL. Normalize scheme to https after resolution if your target site supports it.

The classifier that routes each variant:

from urllib.parse import urljoin, urlsplit

def classify_href(href: str, base: str) -> tuple[str, str | None]:
    """Returns (action, resolved_url). action in {skip, resolve, passthrough}."""
    if not href or not href.strip() or href.strip().startswith("#"):
        return ("skip", None)

    href = href.strip()
    first_segment = href.split("/", 1)[0]

    if first_segment.endswith(":") and not href.startswith("//"):
        scheme = first_segment[:-1].lower()
        if scheme not in ("http", "https"):
            return ("skip", None)
        return ("passthrough", href)

    if href.startswith("//"):
        return ("resolve", urljoin(base, href))

    if href.startswith("http://") or href.startswith("https://"):
        return ("passthrough", href)

    return ("resolve", urljoin(base, href))

One opinionated call here: I skip javascript: and tel: hrefs entirely rather than storing them with a scheme column. Some teams keep them for link-graph completeness. I’ve never once needed them downstream, and they pollute URL-typed columns and break naive validators. If you disagree, store them in a separate table — don’t mix them into your crawl frontier.

Resolving at Scale: Batch-Normalizing Thousands of Scraped Hrefs Before Storage

The pattern that works at scale: resolve every link at ingestion time, against the per-page base, before anything touches the database or queue. No exceptions, no “we’ll fix it in the transform step.”

import logging
from urllib.parse import urljoin, urlsplit

logger = logging.getLogger("link_resolver")

def normalize_page_links(page_url: str, hrefs: list[str]) -> list[str]:
    """Resolve all hrefs from one page against that page's final URL."""
    resolved = []
    for href in hrefs:
        action, url = classify_href(href, page_url)
        if action == "skip":
            continue
        if action in ("resolve", "passthrough"):
            if url is None or not urlsplit(url).netloc:
                logger.warning("unresolvable href=%r base=%r", href, page_url)
                continue
            resolved.append(url)
    return resolved

# --- A crawl batch: two pages, mixed relative and absolute hrefs ---
crawl_batch = [
    {
        "page_url": "https://example.com/listing?page=2",
        "hrefs": [
            "/products/42",
            "/products/43",
            "https://store.example.com/sale",
            "#top",
            "",
            "products/44",
            "mailto:support@example.com",
        ],
    },
    {
        "page_url": "https://store.example.com/listing",
        "hrefs": [
            "/products/42",          # same path, different host — NOT a duplicate
            "//example.com/about",
            "/sale",
        ],
    },
]

all_links = []
for page in crawl_batch:
    links = normalize_page_links(page["page_url"], page["hrefs"])
    all_links.extend(links)
    print(f"{page['page_url']}: {len(links)} storable links")

Output:

https://example.com/listing?page=2: 4 storable links
https://store.example.com/listing: 3 storable links

The before/after on the stored list:

Raw href (as scraped)	Normalized (as stored)
/products/42 (from example.com)	https://example.com/products/42
/products/43	https://example.com/products/43
https://store.example.com/sale	https://store.example.com/sale
products/44	https://example.com/products/44
/products/42 (from store.example.com)	https://store.example.com/products/42
//example.com/about	https://example.com/about
/sale	https://store.example.com/sale
#top, “, mailto:...	(dropped)

Notice the collision that disappears in the right direction: the two /products/42 entries were identical raw strings — a naive dedupe on the raw href would have merged them into one, silently dropping store.example.com from the crawl. After normalization they’re distinct keys for distinct pages. Dedupe on raw hrefs doesn’t just fail to collapse duplicates; it collapses things that aren’t duplicates.

If you’re running this against large batches, the same normalization slot fits naturally into an async batch pipeline — resolve links in the callback or polling handler, before the batch results land in storage. The mechanics of batch jobs and webhooks are covered in Async Scraping at Scale if you want the full pattern.

Canonicalization After Resolution: Sorting Query Params and Collapsing Duplicates

Absolute URLs are necessary but not sufficient for dedupe keys. These two URLs are the same page:

https://example.com/p?id=42&sort=asc
https://example.com/p?sort=asc&id=42

String comparison says no. Your crawl frontier says “two pages,” fetches both, and doubles your spend on half the data. Query parameter order, default ports, and trailing-slash inconsistency all create phantom duplicates.

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def canonical_key(url: str) -> str:
    """Produce a stable dedupe key from an absolute URL."""
    parts = urlsplit(url)

    # Strip default ports
    netloc = parts.netloc
    if netloc.endswith(":80") and parts.scheme == "http":
        netloc = netloc[:-3]
    if netloc.endswith(":443") and parts.scheme == "https":
        netloc = netloc[:-4]

    # Lowercase host (paths and queries are case-sensitive — leave them alone)
    host = netloc.rsplit("@", 1)[-1]
    if "@" in netloc:
        userinfo = netloc.split("@", 1)[0]
        netloc = f"{userinfo}@{host.lower()}"
    else:
        netloc = host.lower()

    # Sort query parameters
    query = urlencode(sorted(parse_qsl(parts.query, keep_blank_values=True)))

    return urlunsplit((parts.scheme, netloc, parts.path, query, ""))

Run it on the sample link set from the previous section, plus the query-param variants:

sample = [
    "https://example.com/p?id=42&sort=asc",
    "https://example.com/p?sort=asc&id=42",
    "https://example.com/products/42",
    "https://example.com:443/products/42",
    "https://EXAMPLE.com/products/42",
]

keys = {canonical_key(u) for u in sample}
print(f"{len(sample)} URLs -> {len(keys)} canonical keys")
# 5 URLs -> 2 canonical keys

Five raw URLs collapse to two canonical keys. On a real crawl of tens of thousands of links, the reduction depends entirely on how inconsistent the site’s internal linking is — but I’ve regularly seen 10–20% of a discovered link set evaporate under canonicalization, and every one of those was a fetch you didn’t have to pay for.

Two things I deliberately do not do in canonical_key: strip trailing slashes and drop tracking parameters. Trailing-slash equivalence (/products/42 vs /products/42/) is site-specific — some servers serve different content. Tracking params (utm_*, ref) are usually safe to drop, but “usually” is doing heavy lifting; some sites genuinely vary content on ref. My recommendation: make both behaviors flags, default off, and enable them per-domain once you’ve verified the target treats those variants as identical. A wrong canonicalization silently drops pages from your crawl, which is worse than fetching a duplicate.

Verifying the Fix: A Regression Test for Your URL Resolver

A resolver without tests is a resolver that will regress. Someone will “simplify” the base URL handling, or a new ingestion path will bypass resolution, and relative URLs will leak back into storage. Lock the behavior in:

import pytest
from urllib.parse import urljoin

# assume resolve_link and canonical_key from earlier sections are importable

@pytest.mark.parametrize("href,base,expected", [
    # absolute-path hrefs resolve against the authority
    ("/products/42", "https://example.com/listing?page=2",
     "https://example.com/products/42"),
    ("/products/42", "https://store.example.com/listing",
     "https://store.example.com/products/42"),
    # directory-relative hrefs resolve against the path directory
    ("products/42", "https://example.com/catalog/listing",
     "https://example.com/catalog/products/42"),
    # protocol-relative inherits the scheme
    ("//store.example.com/x", "https://example.com/listing",
     "https://store.example.com/x"),
    # already absolute passes through unchanged
    ("https://example.com/a", "https://store.example.com/listing",
     "https://example.com/a"),
    # skips
    ("#reviews", "https://example.com/listing", None),
    ("", "https://example.com/listing", None),
    ("mailto:support@example.com", "https://example.com/listing", None),
])
def test_resolve_link(href, base, expected):
    assert resolve_link(href, base) == expected

def test_wrong_base_after_redirect():
    """Regression guard: base must be the FINAL url, not the requested one."""
    requested = "https://example.com/listing"
    final = "https://store.example.com/listing"
    assert resolve_link("/products/42", final) == "https://store.example.com/products/42"
    assert resolve_link("/products/42", final) != urljoin(requested, "/products/42")

@pytest.mark.parametrize("a,b", [
    ("https://example.com/p?id=42&sort=asc", "https://example.com/p?sort=asc&id=42"),
    ("https://example.com/products/42", "https://example.com:443/products/42"),
    ("https://example.com/products/42", "https://EXAMPLE.com/products/42"),
])
def test_canonical_key_collapses(a, b):
    assert canonical_key(a) == canonical_key(b)

def test_canonical_key_preserves_distinct_hosts():
    """Same path on two hosts must NEVER collapse."""
    assert canonical_key("https://example.com/products/42") != \
           canonical_key("https://store.example.com/products/42")

That last test is the one people forget. Canonicalization that collapses across hosts is worse than no canonicalization — it merges distinct pages into one dedupe key and drops data silently.

When you review someone’s link-handling code, run through this checklist:

• Base URL source: Is the base the final response URL (or <base> tag when present), not the original request? Is that documented in the code?
• Empty and anchor hrefs: Are "", whitespace, and #foo explicitly dropped, or do they silently resolve to the current page and cause duplicate fetches?
• Canonical key generation: Does the dedupe key use sorted query params and a lowercased host? Does it not collapse across hosts?
• Logging of unresolvable links: When resolution fails, does the pipeline log the href and base, or swallow it? Unresolvable links are a signal your extractor is picking up junk — you want to see that trend.

Wrap-Up

Relative hrefs are not a parsing detail. They’re a data-integrity bug that corrupts dedupe keys, breaks crawl queues, and makes cross-domain link graphs impossible. The fix is small and the standard library already ships it: urljoin against the correct base — the final response URL — with guards for anchors, empty strings, and non-HTTP schemes, applied at exactly one point in your pipeline, followed by canonicalization for stable dedupe keys.

The discipline matters more than the code. Resolve once, at the ingestion boundary, and test it. Every path that stores a URL must go through the resolver, and the regression tests are what keep the next refactor from quietly reintroducing half-URLs into your dataset.

If you’re rebuilding a pipeline around this, the related posts worth reading next are Turn a Batch of Scraped Pages into a Queryable Dataset for the storage side, and Re-Drive Only the Failures: Handling Partial Batch Results for what to do with the links that still fail after normalization.