How to Scrape Job Postings with Dynamic Filters Using FineData API

How to Scrape Job Postings with Dynamic Filters Using FineData API

Modern job boards and company career pages use dynamic filtering to load content via JavaScript. You can’t just fetch /jobs?q=software+engineer and expect to get all results. The real data comes from XHR requests triggered by filters: location, experience level, remote status, salary range.

This is not a new problem. Even mature scraping stacks struggle with it. Playwright helps. But when you’re building a production-grade job data pipeline, you need more than automation. You need reliability, browser-grade rendering, and structured extraction.

FineData’s API covers this with three features: Playwright rendering, dynamic filter simulation via js_actions, and AI-powered structured extraction. This post shows how to use them together on public listing pages with complex filters — without hammering the origin or inventing authenticated flows.

---

The Problem: Dynamic Filters Break Traditional Scraping

You want to extract all remote software engineering roles at a large tech employer from a public job search page.

The URL is: https://jobs.example.com/search?q=software+engineer&l=remote&jt=fulltime

But the page doesn’t load all jobs in the initial HTML. It makes a fetch request to https://jobs.example.com/api/jobs/collect with query parameters. The response is JSON. The DOM is empty until the request completes.

A naive requests.get() returns an empty result. Even BeautifulSoup fails. You need JavaScript rendering.

Now, try to automate the filter selection. Clicking “Remote” in the UI triggers a state change. The URL updates. The page re-renders. But the new URL doesn’t always reflect every filter — only part of the query string.

This is where most pipelines break. You can’t scrape the results unless you simulate the entire public user flow in a real browser.

---

The Solution: Use js_actions + use_js_render + extract_schema

The POST /api/v1/async/scrape endpoint handles this natively.

Here’s the full working flow:

1. Use use_js_render=true to enable Playwright rendering.
2. Use js_actions to simulate user clicks and form input on the public UI.
3. Wait for the expected element (e.g., #job-listing-container) to appear.
4. Extract structured data using extract_schema with a JSON Schema.
5. Return only the relevant fields.

Step 1: Set Up the Request

import requests
import time

url = "https://api.finedata.ai/api/v1/async/scrape"
headers = {
    "x-api-key": "fd_your_api_key",
    "Content-Type": "application/json"
}

payload = {
    "url": "https://jobs.example.com/search",
    "method": "GET",
    "use_js_render": True,
    "js_wait_for": "selector:#job-listing-container",
    "js_scroll": True,
    "js_actions": [
        {"type": "type", "selector": "input#text-input-what", "value": "software engineer"},
        {"type": "click", "selector": "button#filter-location"},
        {"type": "type", "selector": "input#text-input-where", "value": "remote"},
        {"type": "click", "selector": "button#filter-remote"},
        {"type": "wait", "ms": 3000},
        {"type": "scroll", "direction": "down", "amount": 500}
    ],
    "formats": ["markdown"],
    "extract_schema": {
        "type": "array",
        "items": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "company": {"type": "string"},
                "location": {"type": "string"},
                "posted_date": {"type": "string"},
                "salary": {"type": "string"},
                "job_url": {"type": "string"},
                "job_type": {"type": "string"}
            },
            "required": ["title", "company", "location"]
        }
    },
    "timeout": 30,
    "max_retries": 3,
    "use_antibot": True,
    "tls_profile": "chrome120",
    "use_residential": True,
    "solve_captcha": True,
    "session_id": "job-scraper-public-listings",
    "session_ttl": 1800
}

response = requests.post(url, json=payload, headers=headers)
job_response = response.json()

Step 2: Handle the Async Job

The response returns a job_id and status pending. You now poll:

job_id = job_response['job_id']

while True:
    status_response = requests.get(
        f"https://api.finedata.ai/api/v1/async/jobs/{job_id}",
        headers={"x-api-key": "fd_your_api_key"}
    ).json()

    if status_response['status'] == 'completed':
        result = status_response['result']
        break
    elif status_response['status'] == 'failed':
        raise Exception(f"Job failed: {status_response['error']}")

    time.sleep(2)

Step 3: Extract Structured Data

The result['data']['markdown'] contains the rendered HTML. But the real win is result['data']['json_extracted_data'].

[
  {
    "title": "Software Engineer I",
    "company": "Example Corp",
    "location": "Remote",
    "posted_date": "2026-03-15",
    "salary": "$130k - $160k",
    "job_url": "https://jobs.example.com/view/abc123",
    "job_type": "Full-time"
  }
]

This is production-ready data. No post-processing. No regex. No brittle selectors.

---

Why This Works (And Why It’s Better Than DIY)

1. Playwright Rendering Is Reliable

use_js_render=true runs a real Chrome instance. It waits for the #job-listing-container to appear. It scrolls. It handles XHR responses.

You don’t need to reverse-engineer the site’s private API. You don’t need to parse fetch calls.

2. js_actions Simulates Real User Flow

Clicking the “Remote” filter isn’t just a DOM change. It triggers a state update. The URL changes. The request fires.

js_actions handles this. It’s not just “click a button.” It’s a sequence of actions that mirror what a human would do on the public UI.

This avoids the “ghost click” problem. Some sites only render after a full user gesture sequence.

3. AI-Powered Extraction Beats CSS Selectors

CSS selectors break when the site updates. A single class rename—.job-title → .job-card-title—breaks your entire pipeline.

extract_schema uses an LLM to analyze the page and return data matching your schema. It’s resilient.

For example, if the site uses <div class="job-card"> or <article data-role="job">, the AI adapts.

Honest opinion: I prefer extract_schema over extract_rules because it’s more maintainable. CSS selectors are fragile. LLMs are resilient.

---

Gotchas and Anti-Patterns

1. Don’t Use js_wait_for: "networkidle" on Job Boards

networkidle waits for 500ms with no new network activity. On busy aggregators, the job list might load in 200ms, then a second request for related jobs fires. You’ll get incomplete data.

Use selector:#job-listing-container instead. It’s more deterministic.

2. Avoid use_residential: true on Every Request

It costs +3 tokens. If you’re scraping 1000 job pages per day, that’s 3000 extra tokens per day. At $0.10 per 1000 tokens, that’s $0.30/day.

Use it only when you hit rate limits or empty challenge pages. Otherwise, stick with tls_profile: chrome120.

3. session_id Is for Continuity, Not Login

session_id keeps the same proxy exit across requests. Useful when a public multi-step filter flow needs sticky cookies.

It is not a way to stay logged in. Collecting content behind login or paywall controls is prohibited by our acceptable use policy. Public listings are enough for salary and hiring-trend pipelines.

4. solve_captcha: true Isn’t Free

It costs +10 tokens. And not all sites show a challenge. Use it only when captcha_detected: true.

Check the response for captcha_detected before enabling it. If the challenge still fails, you are not billed for an unsuccessful render on a success-based plan.

---

Personal data and compliance

Job descriptions often include recruiter names and work emails. That is personal data under GDPR Article 6 and CCPA. Document a lawful basis before you store contacts, keep the data controller role on your side, and do not sell scraped email or contact lists — that use is barred by the acceptable use policy. Prefer role, company, location, and skills for market intelligence; treat contact fields as optional and justified.

---

Next Steps

1. Build a Job Board Monitor

Use POST /api/v1/async/batch to scrape multiple public filters in parallel:

• q=software+engineer&l=remote
• q=product+manager&l=New+York
• q=data+scientist&l=San+Francisco

Each job in the batch uses the same js_actions flow. You get all data in one webhook.

2. Automate Data Sync to Your CRM

Use the callback_url to send results to your pipeline. Then use the MCP protocol for AI web data to connect directly to Cursor or Claude Desktop.

Your AI agent can now:

• Read new job postings.
• Extract key fields.
• Suggest outreach messages (only for contacts you have a lawful basis to process).
• Trigger outreach via your CRM.

3. Add Salary Intelligence

Use the same flow to scrape publicly posted salary bands. Track trends over time. Compare large employers against each other using only fields visible without an account.

This is a solid foundation for B2B market intelligence. See our guide on lead generation and B2B data enrichment.

---

Final Thoughts

Scraping job postings with dynamic filters is not just a parsing problem. It is a rendering-fidelity problem.

You can’t win with requests + BeautifulSoup on filter-heavy SPAs.

You need:

• JavaScript rendering (Playwright)
• Action simulation (js_actions)
• Resilient extraction (AI + schema)
• Proxy rotation (residential when needed)
• Inline challenge handling (solve_captcha when a challenge appears)

The API combines Playwright, AI extraction, and proxy rotation in a single request. You don’t need to maintain a fleet of Chrome instances or manage Puppeteer sessions yourself.

Just send the request. Get structured data from public pages. If the page doesn’t render, you don’t pay for failure.

Non-obvious claim: The best web scraping stack for dynamic career UIs isn’t a framework. It’s an API that abstracts browser rendering, filter simulation, and structured extraction.

Use it. You’ll thank yourself when your pipeline doesn’t break after a site update.

And if you’re building a job board scraper for public listings, start here.