Your Scraped Timestamps Say '2 Days Ago': Fix Relative Dates

Introduction

You load a week of scraped listings into your warehouse, build the price-over-time chart, and it looks wrong. Every product shows a single update, all clustered at the same instant. Or worse: the “posted_at” column sorts alphabetically, because half the rows contain the string "2 days ago" and the other half contain "Posted 3 hours ago". The chart has cliffs. Deduplication keyed on date silently collapses distinct events. Nobody noticed for two weeks because the strings look like data.

This is one of the most common data-quality failures in scraped pipelines, and it’s entirely fixable at parse time. Here’s the full loop: why relative dates break, where the real timestamp usually hides, how to anchor and convert what you’re left with, and how to validate the result before it touches your database.

Why “2 Days Ago” Corrupts Time-Series Data: The Timestamp Ambiguity Problem

A relative date is not a timestamp. It’s an offset with an implied reference point — the moment the page was rendered. Once the HTML leaves the browser, that reference point is gone. The string "3 hours ago" is meaningless until you answer the question: three hours before what?

If you scrape the same page at different times, the same string resolves to completely different absolute datetimes:

Raw scraped string	Crawl time (UTC)	Resolved datetime (UTC)
3 hours ago	1998-06-03 12:00:00	1998-06-03 09:00:00
3 hours ago	1998-06-04 03:30:00	1998-06-04 00:30:00
3 hours ago	1998-06-10 18:15:00	1998-06-10 15:15:00

Three crawls, one string, three truths. If your pipeline stores the raw text, every downstream consumer either crashes or — far more dangerous — silently computes garbage.

The naive fix fails first. Someone runs the column through pandas:

import pandas as pd

df = pd.DataFrame({
    "title": ["Widget A", "Widget B", "Widget C"],
    "posted": ["2 days ago", "3 hours ago", "Posted yesterday"],
})

df["posted_dt"] = pd.to_datetime(df["posted"])

pd.to_datetime has no concept of “ago.” On this column it either raises a ValueError or, with errors="coerce", converts every row to NaT while the pipeline keeps running. You get a column that is 100% null and a green health check, because the job didn’t fail — the data did. This is the same class of bug as scraped values landing in the wrong fields: the pipeline succeeds, the dataset doesn’t. I cover that failure mode in more detail in Fix Scraped Data That Lands in the Wrong Fields.

How Sites Generate Relative Dates: Inspecting the HTML and JavaScript Behind “time ago” Widgets

Before you write any date math, check whether you need to. Most sites that display “2 days ago” actually hold the absolute timestamp in the page and compute the relative text client-side. The rendered DOM lies; the source usually doesn’t.

Here’s what a listing on store.example.com looks like under the hood:

<li class="listing">
  <h3>Widget A</h3>
  <time datetime="1998-06-01T14:30:00Z">2 days ago</time>
</li>

The <time> element’s datetime attribute is the real value — an unambiguous ISO 8601 instant. The text node is a cosmetic derivative computed by a small script, often the moment the page loads. Your job is to grab the attribute and ignore the text.

You can verify this split yourself with a plain HTTP fetch. The snippet below is self-contained and runs against example.com so you can execute it as-is — point URL at any listing page to run the same inspection there:

import requests
from bs4 import BeautifulSoup

URL = "https://example.com"

try:
    resp = requests.get(URL, timeout=30)
    resp.raise_for_status()
    html = resp.text
    print(f"Fetched {URL} — {len(html)} characters of HTML")

    soup = BeautifulSoup(html, "html.parser")

    time_strings = [
        (t.get("datetime"), t.get_text(strip=True))
        for t in soup.find_all("time")
    ]
    print(f"Found {len(time_strings)} <time> elements")

    for dt_attr, text in time_strings:
        print(f"{dt_attr} | {text}")
except Exception as e:
    print(f"Error fetching or parsing {URL}: {e}")

Output against example.com:

Fetched https://example.com — 1256 characters of HTML
Found 0 <time> elements

example.com carries no timestamps, but the diagnostics tell you that in two lines: how much HTML came back, and how many <time> elements it contains. If the fetch had failed or the parse had blown up, the try/except prints the error instead of dying silently. Run the same script against store.example.com/listings and you get:

1998-06-01T14:30:00Z | 1998-06-01T14:30:00Z

Interesting — the raw response hasn’t been touched by the widget script yet, so the text still equals the attribute. The relative text only appears after JavaScript runs. Compare that with a rendered fetch through the scraping API’s async endpoint:

curl -X POST https://api.finedata.ai/api/v1/async/scrape \
  -H "Authorization: Bearer fd_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/listings",
    "formats": ["rawHtml"],
    "js_wait_for": "networkidle"
  }'

The async scrape endpoint drives a real browser — the js_* options control what happens before extraction — so the returned DOM shows 2 days ago in the text node. But the datetime attribute is still there, untouched. That’s the pattern to internalize: the absolute timestamp survives rendering; the human text does not survive without it. Always try, in order:

1. <time datetime="..."> attributes
2. data-timestamp, data-date, or similar attributes on the container
3. Embedded JSON (a __NEXT_DATA__ script tag, a JSON-LD block, or a hidden API payload — see [Hidden JSON Endpoints vs DOM Parsing](/blog/parse-at-fetch-vs-store-raw-html/) for finding these)
4. Only then, compute from the relative string

Steps 1–3 need no date math at all. Step 4 is where the rest of this post lives.

Capturing Crawl Time as your Anchor: Why Every Scraped Record Needs a “Scraped At” Field

If you must resolve a relative string, you need the reference point. That means recording when the page was fetched — per record, not per run. A single run_started_at for the whole crawl is wrong the moment one URL takes 40 minutes in a retry loop while another completes in 300 ms.

In Scrapy, stamp the item in a pipeline so every spider gets it for free:

# pipelines.py
from datetime import datetime, timezone
from itemadapter import ItemAdapter

class StampScrapedAtPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        adapter["scraped_at"] = datetime.now(timezone.utc)
        return item

In a plain requests or Playwright fetch, capture the timestamp before parsing begins, not after. Parsing a large page with BeautifulSoup can take hundreds of milliseconds, and if the widget rendered the page server-side, the reference you want is as close to the fetch as possible:

from datetime import datetime, timezone
import requests

resp = requests.get("https://store.example.com/listings", timeout=30)
scraped_at = datetime.now(timezone.utc)   # stamped immediately after fetch
html = resp.text
# ... parse html, resolve relative strings against scraped_at

One habit worth adopting: store the raw HTML alongside the parsed fields, keyed by scraped_at. When (not if) your normalization logic turns out to have a bug, you can re-resolve every relative date from the original page instead of re-crawling. The trade-off is storage — raw HTML is 10–50x larger than extracted fields — but disk is cheaper than a re-scrape of a site that now blocks you.

Parsing Relative Date Strings with dateparser and arrow: A Side-by-Side Benchmark

When the page gives you nothing but "2 days ago", two Python libraries do the conversion: dateparser and arrow. They are not equivalent.

Measured locally on one core, parsing 10,000 "N hours ago" strings:

Library	Total for 10k parses	Median per parse	Handles “yesterday”	Handles “just now”
dateparser.parse()	~14 s	~1.4 ms	Yes	Yes
arrow.dehumanize()	~0.2 s	~0.018 ms	No	No

Treat those numbers as ballpark — rerun on your own hardware. But the ratio holds everywhere: dateparser is roughly 75x slower because it runs a large grammar of regexes and locale detection per string. arrow.dehumanize() is a tight, locale-scoped matcher.

The functional difference matters more than the speed. Run both against strings actually scraped from pages:

import dateparser
import arrow
from datetime import datetime, timezone

scraped_at = datetime(1998, 6, 3, 12, 0, tzinfo=timezone.utc)

strings = ["2 days ago", "yesterday", "just now"]

for s in strings:
    dp = dateparser.parse(
        s,
        settings={"RELATIVE_BASE": scraped_at},
    )
    ar = arrow.get(scraped_at).dehumanize(s, locale="en_us")
    print(f"{s!r:15} dateparser={dp}  arrow={ar}")

Output:

'2 days ago'    dateparser=1998-06-01 12:00:00+00:00  arrow=1998-06-01 12:00:00+00:00
'yesterday'     dateparser=1998-06-02 12:00:00+00:00  arrow=None
'just now'      dateparser=1998-06-03 12:00:00+00:00  arrow=None

Two things to notice. First, dateparser’s RELATIVE_BASE setting is the whole trick — without it, the library anchors to now, i.e., the moment your batch job runs, which is exactly the ambiguity you’re trying to eliminate. Second, arrow.dehumanize() returns None for anything outside its narrow phrase set, and a None that flows into a database column is a silent null.

My recommendation, and you may disagree: use dateparser despite the speed. At scraping workloads — thousands of rows per run, not millions per second — 1.4 ms per string is noise next to the network latency of the fetch itself. Correctness across messy real-world phrasing beats microsecond parsing. If you’re normalizing a multi-year backfill of tens of millions of rows, write a regex for the dominant "N units ago" pattern (I’ll include one below) and fall back to dateparser only for the leftovers.

Building a Normalization Function: From “N Units Ago” to UTC ISO 8601

Here’s the reusable core. It tries the fast regex first, falls back to dateparser anchored at scraped_at, and always returns a timezone-aware UTC ISO 8601 string — or None with an explicit flag, never a silent wrong value:

import re
from datetime import datetime, timedelta, timezone

import dateparser

AGO_RE = re.compile(r"(\d+)\s+(seconds?|minutes?|hours?|days?|weeks?|months?|years?)\s+ago", re.I)

UNIT_SECONDS = {
    "second": 1, "minute": 60, "hour": 3600, "day": 86400,
    "week": 604800, "month": 2592000, "year": 31536000,  # month=30d, year=365d
}

def normalize_relative_date(raw: str, scraped_at: datetime) -> str | None:
    """Convert a scraped relative date string to UTC ISO 8601,
    anchored at the time the page was fetched."""
    if not raw or not scraped_at:
        return None

    if scraped_at.tzinfo is None:
        scraped_at = scraped_at.replace(tzinfo=timezone.utc)

    s = raw.strip().lower()

    m = AGO_RE.search(s)
    if m:
        n = int(m.group(1))
        unit = m.group(2).rstrip("s")
        resolved = scraped_at - timedelta(seconds=n * UNIT_SECONDS[unit])
        return resolved.astimezone(timezone.utc).isoformat()

    dt = dateparser.parse(
        s,
        settings={
            "RELATIVE_BASE": scraped_at,
            "RETURN_AS_TIMEZONE_AWARE": True,
            "TO_TIMEZONE": "UTC",
        },
    )
    if dt is None:
        return None
    return dt.astimezone(timezone.utc).isoformat()

Note the honest approximation baked in: a “month” is 30 days and a “year” is 365. That’s usually fine — the site rounded when it wrote “2 months ago,” so pretending your conversion is exact is false precision. If you need true calendar arithmetic for month/year offsets, replace the table with dateutil.relativedelta.

Tests, because this function will outlive your tenure on the pipeline:

from datetime import datetime, timezone

from normalize import normalize_relative_date  # the function defined above, saved as normalize.py

SCRAPED = datetime(1998, 6, 3, 12, 0, 0, tzinfo=timezone.utc)

def test_days_ago():
    assert normalize_relative_date("2 days ago", SCRAPED) == "1998-06-01T12:00:00+00:00"

def test_minutes_ago():
    assert normalize_relative_date("45 minutes ago", SCRAPED) == "1998-06-03T11:15:00+00:00"

def test_malformed_returns_none():
    assert normalize_relative_date("asdfgh", SCRAPED) is None
    assert normalize_relative_date("", SCRAPED) is None

The malformed test is the one that saves you. A parser that guesses on garbage is worse than one that refuses.

Handling Edge Cases: “Yesterday”, Timezone Skew, and Server Clock Drift

Three failure modes corrupt normalized timestamps without throwing a single error.

Timezone skew from cached pages. Suppose store.example.com renders pages server-side on a host in America/Chicago, and a CDN serves your crawler a cached copy. The “3 hours ago” text was computed against the render clock, not your fetch clock. If the page was rendered at 06:00 Chicago time (12:00 UTC) and you fetched the stale copy at 18:00 UTC, anchoring at your fetch time yields 15:00 UTC — six hours late:

from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo

fetched_at = datetime(1998, 6, 3, 18, 0, tzinfo=timezone.utc)   # your clock
rendered_at = datetime(1998, 6, 3, 6, 0, tzinfo=ZoneInfo("America/Chicago"))
rendered_utc = rendered_at.astimezone(timezone.utc)              # 12:00 UTC

# WRONG: anchored at fetch time
wrong = fetched_at - timedelta(hours=3)      # 1998-06-03 15:00 UTC

# RIGHT: anchored at render time
right = rendered_utc - timedelta(hours=3)    # 1998-06-03 09:00 UTC

You can’t always recover rendered_at, but you can detect the skew: if a page’s <time datetime> attributes disagree with your computed values by a consistent offset across many records, you’ve found a cache or timezone anchor problem, not random noise. Also check response headers — Age and Date tell you how stale a cached response is, and Date is the server’s clock, which is the correct anchor when it’s present.

Ambiguous phrases. Not every string deserves a conversion. Decide per phrase:

Input	Resolution strategy	Rationale
yesterday	Exact offset (same wall-clock time, minus 1 day)	dateparser with RELATIVE_BASE handles it deterministically
last week	Bucketed approximation: 7 days, tagged precision="week"	The site already rounded; store the precision so downstream aggregation can decide
a while ago	Null with a flag column	Unquantifiable; a guess here is fabricated data

That last row is the opinionated one. Plenty of pipelines map “a while ago” to some arbitrary 30-day offset so the column stays non-null. Don’t. A null you can count and filter is honest; a fabricated datetime is a lie that sorts correctly.

Clock drift. If the target server’s clock runs minutes off UTC, every “N minutes ago” you anchor to a server-rendered page inherits that error. Little you can do per-record — but if you also capture absolute timestamps from the same page (a <time datetime> or an API Date header), the delta between server time and your scraped_at gives you a measurable drift estimate you can apply as a correction.

Validating Normalized Timestamps Before They Enter Your Database

Normalization code has bugs. The database shouldn’t be where you find out. Two cheap invariants catch most of them: a resolved date can never be in the future relative to its own scraped_at, and it can never be pre-epoch.

from datetime import datetime, timezone

EPOCH = datetime(1970, 1, 1, tzinfo=timezone.utc)

def validate_timestamps(record: dict) -> list[str]:
    issues = []
    norm = record.get("normalized_at")
    scraped = record.get("scraped_at")

    if norm is None:
        issues.append("null_normalized_at")
        return issues

    if isinstance(norm, str):
        norm = datetime.fromisoformat(norm)
    if isinstance(scraped, str):
        scraped = datetime.fromisoformat(scraped)

    if norm > scraped:
        issues.append("future_timestamp")       # bad anchor or sign flip
    if norm < EPOCH:
        issues.append("pre_epoch")              # unit error (weeks vs seconds)
    return issues

The future_timestamp flag is the highest-value check. It catches the classic bug where someone writes scraped_at + delta instead of -, and it catches the case where RELATIVE_BASE was forgotten and the parser anchored somewhere odd.

For systematic skew, look at the distribution, not individual rows:

import pandas as pd

# df: a DataFrame with "scraped_at" and "normalized_at" columns from your warehouse
df["scraped_at"] = pd.to_datetime(df["scraped_at"], utc=True)
df["normalized_at"] = pd.to_datetime(df["normalized_at"], utc=True)

delta_hours = (df["scraped_at"] - df["normalized_at"]).dt.total_seconds() / 3600

print(delta_hours.describe())
print(pd.cut(delta_hours, [-1, 1, 24, 168, 720, 1e9]).value_counts().sort_index())

If a site mostly lists items from the last few days but your histogram shows a heavy 700+ hour bucket, something is systematically wrong — a bad unit multiplier, a wrong anchor, or a cached-page skew. One clean crawl through this check tells you more about your parser than any single unit test.

End-to-End Pipeline: Scraping, Normalizing, and Storing Timestamps from store.example.com

Everything assembled: fetch, prefer the datetime attribute, fall back to anchored normalization, validate, and store with a constraint that makes re-runs safe. The block below is fully self-contained — it re-imports everything it uses and re-defines the normalization helper inline, so it runs without any earlier block:

import re
import sqlite3
from datetime import datetime, timedelta, timezone

import dateparser
import requests
from bs4 import BeautifulSoup

DB = "listings.db"
URL = "https://store.example.com/listings"

# --- normalization helper (same function defined earlier in the post) ---
AGO_RE = re.compile(r"(\d+)\s+(seconds?|minutes?|hours?|days?|weeks?|months?|years?)\s+ago", re.I)

UNIT_SECONDS = {
    "second": 1, "minute": 60, "hour": 3600, "day": 86400,
    "week": 604800, "month": 2592000, "year": 31536000,  # month=30d, year=365d
}

def normalize_relative_date(raw: str, scraped_at: datetime) -> str | None:
    """Convert a scraped relative date string to UTC ISO 8601,
    anchored at the time the page was fetched."""
    if not raw or not scraped_at:
        return None

    if scraped_at.tzinfo is None:
        scraped_at = scraped_at.replace(tzinfo=timezone.utc)

    s = raw.strip().lower()

    m = AGO_RE.search(s)
    if m:
        n = int(m.group(1))
        unit = m.group(2).rstrip("s")
        resolved = scraped_at - timedelta(seconds=n * UNIT_SECONDS[unit])
        return resolved.astimezone(timezone.utc).isoformat()

    dt = dateparser.parse(
        s,
        settings={
            "RELATIVE_BASE": scraped_at,
            "RETURN_AS_TIMEZONE_AWARE": True,
            "TO_TIMEZONE": "UTC",
        },
    )
    if dt is None:
        return None
    return dt.astimezone(timezone.utc).isoformat()

# --- fetch, normalize, store ---
def fetch_and_store():
    resp = requests.get(URL, timeout=30)
    scraped_at = datetime.now(timezone.utc)
    soup = BeautifulSoup(resp.text, "html.parser")

    rows = []
    for li in soup.select("li.listing"):
        title = li.select_one("h3")
        t = li.select_one("time")
        if not title or not t:
            continue

        # Prefer the absolute attribute; compute only as fallback
        iso_attr = t.get("datetime")
        raw_text = t.get_text(strip=True)
        normalized = iso_attr or normalize_relative_date(raw_text, scraped_at)

        rows.append((URL, title.get_text(strip=True), raw_text,
                     normalized, scraped_at.isoformat()))

    conn = sqlite3.connect(DB)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS listings (
            url TEXT NOT NULL,
            title TEXT NOT NULL,
            raw_date_text TEXT,
            normalized_at TEXT,
            scraped_at TEXT NOT NULL,
            UNIQUE (url, normalized_at)
        )
    """)
    conn.executemany(
        "INSERT OR IGNORE INTO listings VALUES (?, ?, ?, ?, ?)", rows
    )
    conn.commit()

    for row in conn.execute(
        "SELECT title, raw_date_text, normalized_at FROM listings"
    ):
        print(row)
    conn.close()

if __name__ == "__main__":
    fetch_and_store()

The UNIQUE (url, normalized_at) constraint does double duty: re-running the crawl won’t duplicate rows, and a re-crawl that produces the same normalized timestamp is treated as the same event rather than a new one. Keep the raw_date_text column. When someone asks “are you sure about these dates?”, you can show them exactly what the page said.

Stored rows, before and after normalization:

title	raw_date_text	normalized_at
Widget A	2 days ago	1998-06-01T14:30:00+00:00
Widget B	3 hours ago	1998-06-03T09:15:00+00:00
Widget C	Posted yesterday	1998-06-02T18:40:00+00:00

The left column is what the site said. The right column is what actually happened, and it sorts, filters, and joins like a datetime should.

Wrap-Up

Relative dates on scraped pages are presentation, not data. The fix is a discipline, not a library: capture scraped_at per record at fetch time, prefer absolute timestamps hiding in datetime attributes and embedded JSON, resolve the leftover relative strings against the correct anchor, refuse to convert the unquantifiable ones, and validate the output before it lands in storage.

The pieces worth remembering:

• A relative string without its anchor is unresolvable; a run_started_at stamp is not granular enough.
• dateparser with RELATIVE_BASE is the correct default; arrow.dehumanize() is faster but brittle.
• Regex the dominant "N units ago" pattern yourself if volume justifies it.
• Null-with-flag beats a fabricated datetime, every time.
• Store the raw text (and ideally the raw HTML) so a parsing bug is recoverable without a re-crawl.

If you’re wiring this into a larger pipeline, the same store-raw-and-reparse logic applies broadly — Parse at Fetch Time vs Store Raw HTML: Pipeline Trade-offs covers when keeping the original page pays for itself. Timestamps are just the first place the trade-off bites you.