Lead Generation and B2B Data Enrichment: From Web Data to CRM

Lead Generation and B2B Data Enrichment: From Web Data to CRM

Every sale starts with a lead, and the quality of your data directly predicts the quality of your outcomes. A list of 10,000 names with no context is nearly useless. A list of 500 companies with verified contacts, technology stacks, company sizes, and recent growth signals — that’s a pipeline.

The web is the richest source of that data: company directories, professional profiles, industry listings, event attendee lists, job postings, and public business registrations. The challenge isn’t finding it — it’s extracting it systematically, enriching it with context, cleaning it, scoring it, and getting it into a CRM in a usable format. This guide covers the full pipeline, from source identification to CRM sync.

Before you start: this data is almost entirely about individuals in a business capacity — names, titles, work emails. In the EU, that’s still personal data under GDPR Article 6, and even outside the EU most CRMs and email providers expect you to be able to justify why a contact is in your system. Selling scraped email lists or contact databases is prohibited by our acceptable use policy. The Compliance and Ethics section below isn’t an afterthought — read it before you build the pipeline, not after you’ve loaded 50,000 records.

Where to Find Lead and Enrichment Data

Business Directories

Directories are purpose-built lists of companies, often with contact information, industry classification, and company details:

• Industry-specific directories — agency, SaaS, and manufacturing listings curated by vertical
• General business directories — local and regional business listings
• Government registries — SEC EDGAR (public companies), state business registrations, SBA databases
• Chamber of Commerce listings

Professional and Startup Networks

Professional networks and funding databases are high-signal for B2B, but many restrict automated access in their terms and invest heavily in challenge pages. Prefer official APIs and partner programs for profile-level data. For enrichment you control end-to-end, company websites, public directories, and press releases are usually the safer primary sources. Startup ecosystem directories and funding news sites remain useful for stage and investor signals when those pages are public.

Company Websites

The most authoritative source for any single company:

• About / Team page — decision-maker names, titles, headshots
• Contact page — direct email addresses and phone numbers (personal data — keep a lawful basis)
• Careers page — open roles (a growth signal) and technology stack (from job descriptions)
• Blog / News — recent priorities, product launches, partnerships, technology choices
• Footer — social media links, legal entity information

Event, Review, and Community Sites

• Attendee and speaker lists — published lists from trade shows and conferences
• Software review sites — companies discussing competitor products (high-intent leads)
• Developer forums and code hosts — practitioners discussing relevant technologies or problems

Public Records and Technology Detection

• SEC EDGAR, state business registrations, patent databases — financial filings, legal entity information, R&D signals
• HTML fingerprinting — detect tech signatures yourself from public pages (see Step 3), optionally cross-check with third-party detectors

What to Extract

A complete B2B record typically layers three tiers of data on top of a bare company name or domain:

Company-level: name, description, industry vertical, headquarters and office locations, employee count, revenue range, founding year, funding history, technology stack.

Contact-level: decision-maker names and titles, professional email addresses, direct phone numbers, public profile URLs when posted on the company site, role and department. Treat every contact field as personal data.

Signal data: recent news and press mentions, job postings (a growth indicator — see job board scraping for the sourcing side of this), technology changes, review activity, public social links from the company footer.

Field	Source	Priority
Company name	Directory, website	Required
Website URL	Directory, web search	Required
Industry / vertical	Directory, manual classification	High
Company size	Directory, job postings, website	High
Contact name	Team page, directory	High
Job title	Team page	High
Email address	Contact page, pattern detection	Medium
Phone number	Contact page, directory	Medium
Technology stack	Job postings, HTML source detection	Contextual
Funding stage	Funding news, press releases	Contextual
Recent news	Blog, press releases	Contextual

Building the Extraction Pipeline

Step 1: Build or Start From a Seed List

Most pipelines start one of two ways: scraping a directory to discover companies from scratch, or enriching a seed list you already have (a conference attendee export, domains parsed from an existing lead list’s email addresses).

Directories are usually the highest-ROI starting point when you don’t have a seed list — they’re structured, contain many companies per page, and are designed to be browsable:

import requests
import time
from bs4 import BeautifulSoup

FINEDATA_API = "https://api.finedata.ai/api/v1/scrape"
API_KEY = "fd_your_api_key"

def scrape_directory_page(url):
    """Scrape a directory listing page and extract company entries."""
    response = requests.post(
        FINEDATA_API,
        headers={
            "x-api-key": API_KEY,
            "Content-Type": "application/json"
        },
        json={
            "url": url,
            "use_js_render": True,
            "tls_profile": "chrome124",
            "timeout": 30
        }
    )

    if response.status_code != 200:
        return []

    html = response.json()["body"]
    soup = BeautifulSoup(html, "html.parser")
    companies = []

    for listing in soup.select(".company-listing"):
        company = {
            "name": safe_text(listing.select_one(".company-name")),
            "domain": safe_attr(listing.select_one("a.website-link"), "href"),
            "location": safe_text(listing.select_one(".location")),
            "description": safe_text(listing.select_one(".description")),
            "category": safe_text(listing.select_one(".category")),
        }
        if company["name"]:
            companies.append(company)

    return companies


def safe_text(element):
    return element.get_text(strip=True) if element else None

def safe_attr(element, attr):
    return element.get(attr) if element else None

If you’re starting from a seed list instead, the format is just the domains themselves:

seed_companies = [
    {"domain": "acme-corp.com"},
    {"domain": "techstart.io"},
    {"domain": "bigretail.com"},
]

Step 2: Enrich From the Company Website

Once you have a domain, visiting the company’s own pages is the highest-authority way to fill in the record — homepage, about page, team page, and careers page each carry a different slice of the data:

def enrich_from_website(domain):
    """Extract company information from their own website."""
    enriched = {"domain": domain}

    homepage = scrape_page(f"https://{domain}")
    if homepage:
        enriched.update(extract_homepage_data(homepage))
        enriched.update(extract_contact_info(homepage))

    for team_path in ["/team", "/about/team", "/about-us", "/our-team", "/people"]:
        team = scrape_page(f"https://{domain}{team_path}")
        if team and "team" in team.lower():
            enriched["team_members"] = extract_team_members(team)
            enriched.update(extract_contact_info(team))
            break

    for careers_path in ["/careers", "/jobs", "/join-us", "/work-with-us"]:
        careers = scrape_page(f"https://{domain}{careers_path}")
        if careers and ("job" in careers.lower() or "career" in careers.lower()):
            enriched["open_positions"] = count_job_listings(careers)
            break

    return enriched


def scrape_page(url):
    """Scrape a single page and return HTML content, or None on failure."""
    try:
        response = requests.post(
            FINEDATA_API,
            headers={
                "x-api-key": API_KEY,
                "Content-Type": "application/json"
            },
            json={
                "url": url,
                "use_js_render": True,
                "tls_profile": "chrome124",
                "timeout": 20
            }
        )
        if response.status_code == 200:
            return response.json()["body"]
    except Exception:
        pass
    time.sleep(1)
    return None


def extract_homepage_data(html):
    """Extract description and social links from a company homepage."""
    import re
    soup = BeautifulSoup(html, "html.parser")
    data = {}

    meta_desc = soup.find("meta", attrs={"name": "description"})
    if meta_desc:
        data["description"] = meta_desc.get("content", "")

    social_patterns = {
        "professional_network": r"(?:directory|profiles)\.example\.com/company/[\w-]+",
        "twitter": r"(?:twitter|x)\.com/[\w]+",
        "github": r"github\.com/[\w-]+",
    }
    page_text = str(soup)
    for platform, pattern in social_patterns.items():
        match = re.search(pattern, page_text)
        if match:
            data[f"{platform}_url"] = f"https://{match.group()}"

    return data


def extract_team_members(html):
    """Extract team member names and titles from a team page."""
    soup = BeautifulSoup(html, "html.parser")
    members = []

    for card in soup.select(
        "[class*='team-member'], [class*='person'], [class*='staff'], "
        "[class*='leadership'], [class*='executive']"
    ):
        name_el = card.select_one("h2, h3, h4, [class*='name']")
        title_el = card.select_one("p, span, [class*='title'], [class*='role'], [class*='position']")
        if name_el:
            member = {"name": name_el.get_text(strip=True)}
            if title_el:
                member["title"] = title_el.get_text(strip=True)
            members.append(member)

    return members


def extract_contact_info(html):
    """Extract emails and phone numbers from a page's HTML."""
    import re
    data = {}

    emails = re.findall(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}", html)
    filtered_emails = [
        e for e in emails
        if not any(prefix in e.lower() for prefix in
            ["noreply", "no-reply", "support", "info@", "admin@", "webmaster"])
    ]
    if filtered_emails:
        data["emails"] = list(set(filtered_emails))

    phones = re.findall(r"[\+]?1?[-.\s]?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}", html)
    if phones:
        data["phone"] = phones[0].strip()

    return data

Step 3: Detect the Technology Stack

Knowing what tools a company runs helps with both relevance scoring and personalization — it’s a light HTML fingerprint, not an intrusive scan:

def detect_technology_stack(html):
    """Detect technologies used by analyzing the HTML source."""
    technologies = []

    tech_signatures = {
        "React": ["react", "_reactRootContainer", "__NEXT_DATA__"],
        "Vue.js": ["vue", "__vue__", "vue-router"],
        "WordPress": ["wp-content", "wp-includes"],
        "Shopify": ["cdn.shopify.com", "Shopify.theme"],
        "HubSpot": ["hs-scripts.com", "hubspot"],
        "Google Analytics": ["google-analytics.com", "gtag"],
        "Intercom": ["intercom", "intercomSettings"],
        "Segment": ["cdn.segment.com", "analytics.js"],
        "Stripe": ["js.stripe.com", "stripe"],
        "Salesforce": ["force.com", "salesforce"],
    }

    html_lower = html.lower()
    for tech, signatures in tech_signatures.items():
        if any(sig.lower() in html_lower for sig in signatures):
            technologies.append(tech)

    return technologies

Data Cleaning, Deduplication, and Scoring

Raw scraped data is messy, and merging records from multiple sources multiplies the mess. The same company can show up as “Acme Corp”, “Acme Corporation”, “ACME Corp.”, and acme-corp.com — before it goes into your CRM, it needs to go through cleaning, deduplication, and scoring.

Deduplication

Domain matching is the highest-confidence signal; fuzzy name matching catches the rest:

from difflib import SequenceMatcher

def normalize_domain(domain):
    """Normalize a domain for comparison."""
    domain = domain.lower().strip()
    domain = domain.replace("https://", "").replace("http://", "")
    domain = domain.replace("www.", "")
    return domain.rstrip("/")


def normalize_company_name(name):
    """Normalize company name for matching."""
    name = name.lower().strip()
    suffixes = [" inc", " inc.", " llc", " ltd", " ltd.", " corp", " corp.",
                " co.", " company", " gmbh", " ag", " sa"]
    for suffix in suffixes:
        if name.endswith(suffix):
            name = name[:-len(suffix)].strip()
    return name


def is_duplicate(company_a, company_b, threshold=0.85):
    """Determine if two company records refer to the same company."""
    domain_a = normalize_domain(company_a.get("domain", ""))
    domain_b = normalize_domain(company_b.get("domain", ""))
    if domain_a and domain_b and domain_a == domain_b:
        return True

    name_a = normalize_company_name(company_a.get("name", ""))
    name_b = normalize_company_name(company_b.get("name", ""))
    if name_a and name_b:
        similarity = SequenceMatcher(None, name_a, name_b).ratio()
        if similarity >= threshold:
            return True

    return False

Data Standardization

• Company names — strip Inc., LLC, Ltd. suffixes for matching; keep them in the CRM record itself
• Phone numbers — normalize to E.164 format
• Addresses — parse into structured components (street, city, state, zip)
• Job titles — map variations to standard titles (VP of Sales, Vice President Sales, Head of Sales → “VP Sales”)
• URLs — normalize (remove trailing slashes, www. prefix, protocol differences)

Email Verification

Don’t load unverified emails into your CRM — check syntax validity, verify the domain has MX records, and run addresses through an email verification service for bounce checking before sending anything. Loading bad emails damages your sender reputation and skews your CRM data quality.

Data Quality Scoring

This measures how complete a record is — independent of whether the company is a good sales fit:

def calculate_quality_score(company):
    """Score enriched data completeness from 0-100."""
    score = 0

    core_fields = {
        "description": 10, "employee_count": 10, "industry": 10,
        "location": 10, "founded_year": 5, "revenue_range": 5,
    }
    for field, points in core_fields.items():
        if company.get(field):
            score += points

    if company.get("team_members"):
        score += 15
    if company.get("emails"):
        score += 10
    if company.get("phone"):
        score += 5

    if company.get("technologies"):
        score += 5
    if company.get("open_positions") is not None:
        score += 5
    if company.get("professional_network_url"):
        score += 5
    if company.get("recent_news"):
        score += 5

    return min(score, 100)

Lead Scoring

This is a different question: not “how complete is this record” but “how good a fit is this company for what we sell.” Build it around your ideal customer profile:

def score_lead(company):
    """Score how well a company fits your ideal customer profile."""
    score = 0

    size = company.get("employee_count", 0)
    if 50 <= size <= 500:
        score += 30
    elif 500 < size <= 2000:
        score += 20
    elif 10 <= size < 50:
        score += 10

    target_industries = ["saas", "ecommerce", "fintech", "marketing"]
    if company.get("industry", "").lower() in target_industries:
        score += 25

    if company.get("emails"):
        score += 15
    if company.get("phone"):
        score += 10

    if company.get("open_positions"):
        score += 10
    if company.get("recent_funding"):
        score += 10

    return score

In practice you’ll filter on quality score first (is the record even usable?) and rank on lead score second (is it worth a rep’s time?).

CRM Integration

The final step is getting clean, scored, deduplicated records into your sales team’s hands. Most CRMs offer APIs for programmatic lead creation — Salesforce (REST or Bulk API), HubSpot (Contacts API), Pipedrive, and Close all have well-documented options.

A general sync pattern handles create/update/skip logic based on the quality score:

from datetime import datetime, timezone

def sync_to_crm(enriched_companies, crm_client, min_quality=30):
    """Sync enriched data to CRM, creating or updating records."""
    results = {"created": 0, "updated": 0, "skipped": 0}

    for company in enriched_companies:
        quality = calculate_quality_score(company)
        if quality < min_quality:
            results["skipped"] += 1
            continue

        existing = crm_client.find_company(domain=company["domain"])
        if existing:
            crm_client.update_company(existing["id"], {
                "enrichment_score": quality,
                "technologies": ", ".join(company.get("technologies", [])),
                "open_positions": company.get("open_positions", 0),
                "last_enriched": datetime.now(timezone.utc).isoformat()
            })
            results["updated"] += 1
        else:
            crm_client.create_company({
                "name": company.get("name", company["domain"]),
                "domain": company["domain"],
                "description": company.get("description", ""),
                "industry": company.get("industry", ""),
                "employee_count": company.get("employee_count"),
                "enrichment_score": quality,
                "lead_source": "web_enrichment",
                "technologies": ", ".join(company.get("technologies", [])),
            })
            results["created"] += 1

    return results

For a single CRM, a direct API call is often simpler than a generic client abstraction — here’s the same idea against HubSpot’s Contacts API:

def push_to_hubspot(lead, hubspot_api_key):
    """Create or update a contact in HubSpot."""
    contact_data = {
        "properties": {
            "company": lead["name"],
            "website": lead.get("website", ""),
            "email": lead.get("emails", [None])[0],
            "phone": lead.get("phone", ""),
            "industry": lead.get("industry", ""),
            "lead_source": "web_scraping",
            "lead_score": lead.get("score", 0)
        }
    }

    response = requests.post(
        "https://api.hubapi.com/crm/v3/objects/contacts",
        headers={
            "Authorization": f"Bearer {hubspot_api_key}",
            "Content-Type": "application/json"
        },
        json=contact_data
    )

    return response.status_code == 201

Best practices for CRM loading: map fields carefully so scraped data lands in the right places, tag the lead source (“Web Enrichment”) so you can track conversion by source, check for existing records before creating new ones (domain match first, then fuzzy name), and add a note about where and why the lead was found.

Maintaining Data Quality Over Time

Enrichment isn’t a one-time activity. People change jobs, companies move offices, phone numbers change. Plan for ongoing maintenance:

• Re-enrich on a schedule. Run the pipeline against existing records every 3-6 months.
• Monitor bounce rates. A rising bounce rate on a segment is a signal it’s time to re-enrich those records specifically.
• Track freshness. Add a last_enriched timestamp to every record and flag records past your staleness threshold.
• Decay the score. Automatically lower quality scores over time for records that haven’t been refreshed, so stale-but-complete records don’t outrank recently-verified ones.

Compliance and Ethics

Lead generation and enrichment with public web data sits in a space that rewards being deliberate about the rules, not just fast. A scraping API is infrastructure: lawful basis, ToS compliance for each target, and what you do with contacts are your responsibility as the data controller — see acceptable use.

GDPR (European Union)

Even B2B contact data — a work email, a job title — counts as personal data under GDPR when it identifies an individual. Article 6 requires a lawful basis before you store or use it:

• You need a legitimate interest (or another Article 6 basis) for processing it, and that basis has to be documented, not assumed
• Individuals have the right to access, rectify, and delete their data
• You must provide a clear way for people to opt out, and honor it promptly
• Processing must be proportionate to the purpose — collecting a title and work email for B2B outreach is a very different proportionality question than collecting someone’s full browsing history

CCPA (California)

California residents have rights over their personal information regardless of B2B context — the “it’s a business contact, not a consumer” distinction that some GDPR guidance allows does not carry over to CCPA in the same way.

CAN-SPAM (United States)

If you’re emailing leads: include a clear unsubscribe mechanism, don’t use misleading subject lines or sender information, and include your physical address.

Our AUP and reseller limits

Selling scraped email lists, contact databases, or similar personal-data products is strictly prohibited under the platform acceptable use policy. Build pipelines for your own outbound and CRM enrichment — not for packaging and reselling contact dumps.

General Practices That Apply Regardless of Jurisdiction

• Only collect data that’s publicly available — don’t scrape behind login walls or access private databases
• Respect robots.txt and each source’s terms of service
• Maintain a suppression list for anyone who asks to be removed, and check it before every send or sync
• Practice data minimization — collect what your sales process actually uses, not everything you technically can
• Document your data collection and processing procedures so you can answer a subject access request without a scramble
• Consult a lawyer if you’re operating in regulated industries or across borders — this section is an engineering-level orientation, not legal advice

Putting It All Together

A complete lead generation and enrichment pipeline looks like this:

1. Source identification — map directories, websites, and databases relevant to your ICP
2. Extraction — use FineData’s API to scrape structured data from directories and company websites at scale
3. Enrichment — layer on technology stack, team, and growth signals from each company’s own site
4. Cleaning and deduplication — standardize fields and merge duplicate records (domain match first, fuzzy name second)
5. Scoring — a quality score for completeness, a lead score for ICP fit
6. CRM sync — push qualified records into your sales workflow, skipping anything below your quality bar
7. Maintenance — re-enrich on a schedule so scores and contact details don’t silently go stale
8. Feedback loop — track which leads convert and use it to refine both scoring models

The teams that generate the most pipeline from web data aren’t just scraping more — they’re scraping smarter, enriching what they find, and treating compliance as part of the pipeline instead of an afterthought.

Get started with a free account and turn the open web into a scored, enriched, CRM-ready lead source.