Main Article Title
This is the introduction paragraph with bold text and italics.
Subheading
- List item one
- List item two
This is a blockquote from the articleLink text
sh Code Example
You are unable to access g2.com
``` ## Why maintaining anti-blocking yourself is costly Each detection layer requires a different countermeasure, and each countermeasure requires ongoing maintenance. IP rotation means sourcing, validating, and retiring proxies. TLS fingerprinting means patching your HTTP client to randomize handshake signatures, something most libraries don't support natively. Browser fingerprinting means keeping headless browser patches current as detection systems update weekly. CAPTCHA solving means integrating third-party solvers and handling failures. In practice, teams that build this in-house find that anti-blocking maintenance consumes a significant share of their engineering effort, often more time than building the actual data pipeline. When a target site updates its defenses, your scraper fails silently, returning empty fields or stale data while appearing to work normally. By the time you notice, your downstream systems have been ingesting bad data for hours or days. Every Bright Data scraping product shares the same anti-blocking engine. The difference between products is what you get back and how much control you keep. ## How Bright Data handles each detection layer **IP reputation.** Requests are routed through residential IPs from real ISPs across 195+ countries. Anti-bot systems treat these as regular household traffic. IPs are automatically rotated per request and flagged addresses are retired from the pool. **TLS fingerprinting.** When your HTTP client opens a connection, it sends a TLS Client Hello that anti-bot systems hash into a JA3 fingerprint. Python `requests`, for example, always produces the same hash, which is a known bot signature. Bright Data generates a unique TLS fingerprint per request, matching the diversity of real browser populations. **Browser fingerprinting.** Sites collect Canvas renders, WebGL data, installed fonts, screen dimensions, and audio context to build a device profile. Bright Data emulates complete, consistent browser environments that pass checks from Cloudflare Turnstile, Akamai Bot Manager, and other major detection systems. **Behavioral analysis.** Anti-bot systems track request timing, navigation sequences, and interaction patterns. Bright Data varies request timing, simulates realistic navigation patterns, and manages session state to match human browsing behavior. **CAPTCHA solving.** CAPTCHAs are solved automatically, including reCAPTCHA, hCaptcha, Cloudflare Turnstile, and others. You never see the challenge. The response arrives as if no CAPTCHA existed. ## Which product fits your situation Each product inherits the anti-blocking capabilities described above. The difference is what you get back and how much control you need. **I want full control over my HTTP client and just need IPs that won't get flagged.** Use [Bright Data Proxies](/proxy-networks/introduction). Four types are available: residential (highest trust, best for protected sites), datacenter (fastest, best for unprotected sites), ISP (residential trust at datacenter speed), and mobile (highest trust, best for heavily protected targets). Proxies solve IP reputation but leave the other four detection layers to you. **I need raw HTML from protected sites without building anti-blocking myself.** Use [Web Unlocker](/scraping-automation/web-unlocker/introduction). Send a URL, get back clean HTML. All five detection layers are handled in a single API call: IP rotation, TLS fingerprinting, browser fingerprinting, behavioral emulation, and CAPTCHA solving. **I need to navigate to click through pagination, or interact with JavaScript-heavy pages.** Use [Bright Data Browser API](/scraping-automation/scraping-browser/introduction). Full cloud-hosted browsers controlled through Puppeteer or Playwright with all anti-blocking built in. Unlike Web Unlocker, Browser API runs a real GUI browser (not headless), which produces authentic rendering artifacts that detection systems are far less likely to flag. **I need search engine results from Google, Bing, or other engines without getting blocked.** Use [SERP API](/scraping-automation/serp-api/introduction). Search engines are among the most aggressively protected targets. SERP API handles all the anti-blocking and returns structured JSON with organic results, ads, featured snippets, and knowledge panels already parsed. **I want structured data from popular websites without writing or maintaining parsers.** Use [Web Scraper API](/datasets/scrapers/scrapers-library/overview). 650+ pre-built scrapers for sites like Amazon, LinkedIn, Instagram, YouTube, TikTok, and Google Maps. You get clean JSON with an average of 220+ data fields per scraper. When target sites change, Bright Data updates the scrapers. ## Common misconceptions **"Residential proxies alone solve blocking."** They handle IP reputation, but modern systems also check TLS fingerprints, browser fingerprints, and behavioral patterns. Proxies are one layer out of five. **"Stealth plugins make headless browsers undetectable."** Cloudflare Turnstile and Akamai Bot Manager detect patched headless browsers even with Playwright Stealth applied. Browser API solves this by running a real GUI browser rather than a patched headless one. **"Slowing down requests prevents blocks."** Rate limiting helps avoid basic IP bans, but sites like LinkedIn, Instagram, and Amazon use session-level fingerprinting that detects automation regardless of speed. **"Proxies and Web Unlocker do the same thing."** Proxies route your requests through different IPs only. Web Unlocker also manages TLS fingerprints, solves CAPTCHAs, emulates browser behavior, handles JavaScript rendering, and retries with fresh fingerprints on failure. ## FAQs"data\_format": "markdown" to receive Markdown SERP structure.
"brd\_json=1" to receive JSON SERP structure.
response_id>}>
Use the `response_id` from the webhook to fetch the actual result:
```sh wrap theme={null}
curl --silent --compressed \
"https://api.brightdata.com/unblocker/get_result?response_id=Send a URL, receive structured JSON/CSV. No proxies, browsers, anti-bot systems or parsing to manage. 1000+ pre-built scrapers cover LinkedIn, Instagram, TikTok, Amazon, Google Maps and more.
This is the introduction paragraph with bold text and italics.
This is a blockquote from the articleLink text
sh Code Example
Connect to our MCP clients for seamless integration.
` | ISO country code for geo-targeting (e.g. `us`, `de`, `jp`) |
| `--zone ` | Web Unlocker zone name |
| `--mobile` | Use a mobile user agent |
| `--async` | Submit async, return a snapshot ID |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash Markdown (default) theme={null}
brightdata scrape https://news.ycombinator.com
```
```bash HTML theme={null}
brightdata scrape https://example.com -f html
```
```bash JSON with geo-targeting theme={null}
brightdata scrape https://amazon.com -f json --country us -o product.json
```
```bash Screenshot theme={null}
brightdata scrape https://example.com -f screenshot -o page.png
```
```bash Async mode theme={null}
brightdata scrape https://example.com --async
```
```bash Pipe to markdown reader theme={null}
brightdata scrape https://docs.github.com | glow -
```
***
## `brightdata search `
Search Google, Bing, or Yandex via Bright Data's SERP API. Google returns structured JSON with organic results, ads, People Also Ask, and related searches. Bing and Yandex return markdown by default.
| Flag | Description |
| --------------------- | --------------------------------------------- |
| `--engine ` | `google` (default), `bing`, `yandex` |
| `--country ` | Localized results (e.g. `us`, `de`) |
| `--language ` | Language code (e.g. `en`, `fr`) |
| `--page ` | Page number, 0-indexed (default: `0`) |
| `--type ` | `web` (default), `news`, `images`, `shopping` |
| `--device ` | `desktop`, `mobile` |
| `--zone ` | SERP zone name |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash Basic search theme={null}
brightdata search "typescript best practices"
```
```bash Localized search theme={null}
brightdata search "restaurants berlin" --country de --language de
```
```bash News search theme={null}
brightdata search "AI regulation" --type news
```
```bash Pagination theme={null}
brightdata search "web scraping" --page 1
```
```bash Extract URLs with jq theme={null}
brightdata search "open source scraping" --json | jq -r '.organic[].link'
```
```bash Bing search theme={null}
brightdata search "bright data pricing" --engine bing
```
***
## `brightdata discover `
AI-powered web discovery. Submit a query with optional intent, and Bright Data finds, ranks, and optionally extracts full-page content for each result.
| Flag | Description |
| --------------------------- | ---------------------------------------------------- |
| `--intent ` | AI intent to evaluate and rank result relevance |
| `--country ` | ISO country code (default: `US`) |
| `--city ` | City for localized results (e.g. `"New York"`) |
| `--language ` | Language code (default: `en`) |
| `--num-results ` | Number of results to return |
| `--filter-keywords ` | Comma-separated keywords that must appear in results |
| `--include-content` | Include full page content (markdown) in each result |
| `--no-remove-duplicates` | Keep duplicate results |
| `--start-date ` | Only content updated from date (`YYYY-MM-DD`) |
| `--end-date ` | Only content updated until date (`YYYY-MM-DD`) |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output (raw / indented) |
```bash Basic discovery theme={null}
brightdata discover "AI trends"
```
```bash With AI intent for relevance ranking theme={null}
brightdata discover "AI trends" \
--intent "Prioritize institutional reports for VC research"
```
```bash Include full page content theme={null}
brightdata discover "AI trends" --include-content --num-results 5
```
```bash Geo-targeted with date range theme={null}
brightdata discover "best restaurants" --country US --city "New York" \
--start-date 2025-01-01 --end-date 2025-12-31
```
```bash Filter by keywords theme={null}
brightdata discover "generative AI SaaS" --filter-keywords "revenue,SaaS"
```
```bash JSON output to file theme={null}
brightdata discover "AI trends" --num-results 10 --pretty -o results.json
```
For best results with `--intent`, use a structured formula: describe your persona, what to prioritize, the depth of analysis, and what to exclude. See the [Discover API reference](/api-reference/discover) for detailed guidance.
***
## `brightdata pipelines [params...] [options]`
Extract structured data from 40+ platforms. Triggers an async collection job, polls until results are ready, and returns the data.
| Flag | Description |
| --------------------- | ------------------------------------------ |
| `--format ` | `json` (default), `csv`, `ndjson`, `jsonl` |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash theme={null}
# List all available pipeline types
brightdata pipelines list
```
```bash LinkedIn profile theme={null}
brightdata pipelines linkedin_person_profile "https://linkedin.com/in/username"
```
```bash Amazon product theme={null}
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB"
```
```bash Amazon search theme={null}
brightdata pipelines amazon_product_search "laptop" "https://amazon.com"
```
```bash Instagram profile theme={null}
brightdata pipelines instagram_profiles "https://instagram.com/username"
```
```bash YouTube comments theme={null}
brightdata pipelines youtube_comments "https://youtube.com/watch?v=..." 50
```
```bash CSV export theme={null}
brightdata pipelines amazon_product "https://amazon.com/dp/..." --format csv -o product.csv
```
### Supported platforms
| Type | Platform | Parameters |
| ------------------------ | ---------------------- | ------------------------ |
| `amazon_product` | Amazon product page | `` |
| `amazon_product_reviews` | Amazon reviews | `` |
| `amazon_product_search` | Amazon search results | ` ` |
| `walmart_product` | Walmart product page | `` |
| `walmart_seller` | Walmart seller profile | `` |
| `ebay_product` | eBay listing | `` |
| `bestbuy_products` | Best Buy | `` |
| `etsy_products` | Etsy | `` |
| `homedepot_products` | Home Depot | `` |
| `zara_products` | Zara | `` |
| `google_shopping` | Google Shopping | `` |
| Type | Platform | Parameters |
| -------------------------- | ---------------------- | -------------------------------- |
| `linkedin_person_profile` | LinkedIn person | `` |
| `linkedin_company_profile` | LinkedIn company | `` |
| `linkedin_job_listings` | LinkedIn jobs | `` |
| `linkedin_posts` | LinkedIn posts | `` |
| `linkedin_people_search` | LinkedIn people search | ` ` |
| `crunchbase_company` | Crunchbase | `` |
| `zoominfo_company_profile` | ZoomInfo | `` |
| Type | Platform | Parameters |
| ------------------------------- | -------------------- | ---------------------- |
| `instagram_profiles` | Instagram profiles | `` |
| `instagram_posts` | Instagram posts | `` |
| `instagram_reels` | Instagram reels | `` |
| `instagram_comments` | Instagram comments | `` |
| `facebook_posts` | Facebook posts | `` |
| `facebook_marketplace_listings` | Facebook Marketplace | `` |
| `facebook_company_reviews` | Facebook reviews | ` [num_reviews]` |
| `facebook_events` | Facebook events | `` |
| `tiktok_profiles` | TikTok profiles | `` |
| `tiktok_posts` | TikTok posts | `` |
| `tiktok_shop` | TikTok shop | `` |
| `tiktok_comments` | TikTok comments | `` |
| `x_posts` | X (Twitter) posts | `` |
| `youtube_profiles` | YouTube channels | `` |
| `youtube_videos` | YouTube videos | `` |
| `youtube_comments` | YouTube comments | ` [num_comments]` |
| `reddit_posts` | Reddit posts | `` |
| Type | Platform | Parameters |
| --------------------------- | ----------------------- | -------------------- |
| `google_maps_reviews` | Google Maps reviews | ` [days_limit]` |
| `google_play_store` | Google Play | `` |
| `apple_app_store` | Apple App Store | `` |
| `github_repository_file` | GitHub repository files | `` |
| `yahoo_finance_business` | Yahoo Finance | `` |
| `zillow_properties_listing` | Zillow | `` |
| `booking_hotel_listings` | Booking.com | `` |
Run `brightdata pipelines list` in your terminal to see all available types at any time.
***
## `brightdata scraper`
Build, run, and maintain custom [Bright Data Scraper Studio](/datasets/scraper-studio/build-with-the-cli) scrapers from the terminal. Each scraper is identified by a Collector ID (a `c_*` string) that stays stable across runs and self-healing.
### `brightdata scraper create `
Build a scraper from a natural-language description using Bright Data's AI Agent. Returns a Collector ID.
```bash theme={null}
brightdata scraper create https://news.ycombinator.com \
"Extract top stories: title, url, points, author, comment count"
```
AI generation typically takes 5 to 15 minutes and can run up to 25 minutes on complex targets.
### `brightdata scraper run [url]`
Run a scraper on one or more URLs and return the data. The CLI tries real-time mode first and falls back to batch mode automatically when a run exceeds the real-time page-load limit.
| Flag | Description |
| --------------------- | ------------------------ |
| `-o, --output ` | Write output to file |
| `--json` | Force JSON output |
| `--pretty` | Pretty-print JSON output |
```bash theme={null}
brightdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --pretty
```
### `brightdata scraper heal `
Fix an existing scraper in place via AI self-healing. The Collector ID does not change. By default, `heal` stops at an approval gate and returns `status: "awaiting_approval"` with a `preview_result`.
| Flag | Description |
| --------------------- | ---------------------------------------------------------------------- |
| `--url ` | Verify target woven into the next-step hint. Not sent to the heal call |
| `--auto-approve` | Approve automatically at the gate and poll through to `done` |
| `--timeout ` | Polling timeout (default: `600`) |
| `--max-retries ` | Max retries on the AI-Flow concurrent-job 429 cap (default: `4`) |
| `--no-retry` | Fail immediately on 429 instead of waiting through the cap |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output (raw / indented) |
```bash theme={null}
brightdata scraper heal c_mpohus372o5tmid1jk \
"The price field returns null since the redesign. Re-capture price and currency." \
--url https://example.com/product/1
```
### `brightdata scraper approve `
Commit a heal that is awaiting approval, or reject it with `--reject`.
| Flag | Description |
| --------------------- | ------------------------------------------------------ |
| `--reject` | Reject the proposed fix instead of approving it |
| `--url ` | Verify target woven into the next-step hint on success |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output |
```bash theme={null}
brightdata scraper approve c_mpohus372o5tmid1jk --url https://example.com/product/1
brightdata scraper approve c_mpohus372o5tmid1jk --reject
```
The self-healing flow is run, inspect, `heal`, `approve`, re-run. See the [Scraper Studio CLI quickstart](/datasets/scraper-studio/build-with-the-cli) for a full walkthrough. `scraper heal` and `scraper approve` were added in CLI v0.3.1.
***
## `brightdata status `
Check the status of an async snapshot job (from `--async` scrapes or pipeline collections).
| Flag | Description |
| --------------------- | -------------------------------- |
| `--wait` | Poll until the job completes |
| `--timeout ` | Polling timeout (default: `600`) |
| `-o, --output ` | Write output to file |
| `--json` / `--pretty` | JSON output |
```bash theme={null}
brightdata status s_abc123xyz
brightdata status s_abc123xyz --wait --pretty
brightdata status s_abc123xyz --wait --timeout 300
```
***
## `brightdata browser`
Control a real browser session powered by Bright Data's [Scraping Browser](/scraping-automation/scraping-browser). A lightweight local daemon holds the browser connection open between commands, giving you persistent state without reconnecting on every call.
```bash theme={null}
brightdata browser [options]
```
### Global flags
These flags work with every `browser` subcommand:
| Flag | Description |
| --------------------- | ------------------------------------------------------------------------------------ |
| `--session ` | Session name for running multiple isolated sessions in parallel (default: `default`) |
| `--country ` | Geo-target by ISO country code. On `open`, changing country reconnects the browser |
| `--zone ` | Scraping Browser zone (default: `cli_browser`) |
| `--timeout ` | IPC command timeout in milliseconds (default: `30000`) |
| `--idle-timeout ` | Daemon auto-shutdown after idle (default: `600000` / 10 min) |
| `--json` / `--pretty` | JSON output |
| `-o, --output ` | Write output to file |
### Subcommands
Navigate to a URL. Starts the daemon and browser session automatically if not already running.
```bash theme={null}
brightdata browser open
brightdata browser open https://amazon.com --country us --session shop
```
| Flag | Description |
| --------------------- | ----------------------------------------------------------------------------------- |
| `--country ` | Geo-targeting. Reconnects the browser if the country changes on an existing session |
| `--zone ` | Browser zone name |
| `--idle-timeout ` | Daemon idle timeout for this session |
Capture the page as a text accessibility tree. This is the primary way AI agents read page content - far more token-efficient than raw HTML.
Each interactive element gets a `ref` (e.g. `e1`, `e2`) that you pass to `click`, `type`, `fill`, and other interaction commands.
```bash theme={null}
brightdata browser snapshot
brightdata browser snapshot --compact # Interactive elements + ancestors only
brightdata browser snapshot --interactive # Interactive elements as a flat list
brightdata browser snapshot --depth 3 # Limit tree depth
brightdata browser snapshot --selector "main" # Scope to a CSS subtree
```
**Example output:**
```text theme={null}
Page: Example Domain
URL: https://example.com
- heading "Example Domain" [level=1]
- paragraph "This domain is for use in illustrative examples."
- link "More information..." [ref=e1]
```
| Flag | Description |
| ------------------ | ------------------------------------------------------------------------------- |
| `--compact` | Only interactive elements and their ancestors (70-90% fewer tokens) |
| `--interactive` | Only interactive elements, as a flat list |
| `--depth ` | Limit tree depth |
| `--selector ` | Scope snapshot to elements matching a CSS selector |
| `--wrap` | Wrap output in content boundaries (useful for AI agent prompt injection safety) |
Capture a PNG screenshot of the current viewport.
```bash theme={null}
brightdata browser screenshot
brightdata browser screenshot ./result.png
brightdata browser screenshot --full-page -o page.png
brightdata browser screenshot --base64
```
| Flag | Description |
| ------------- | ---------------------------------------------------------- |
| `[path]` | Where to save the PNG (default: temp directory) |
| `--full-page` | Capture the full scrollable page, not just the viewport |
| `--base64` | Output base64-encoded PNG data instead of saving to a file |
Interact with elements using their snapshot `ref` values.
```bash theme={null}
# Click an element
brightdata browser click e3
# Type text into a field (clears first by default)
brightdata browser type e5 "search query"
brightdata browser type e5 " more text" --append # Append to existing value
brightdata browser type e5 "search query" --submit # Press Enter after typing
# Fill a form field directly (no keyboard simulation)
brightdata browser fill e2 "user@example.com"
# Select a dropdown option by visible label
brightdata browser select e4 "United States"
# Check / uncheck a checkbox or radio button
brightdata browser check e7
brightdata browser uncheck e7
# Hover over an element
brightdata browser hover e2
```
| Flag (for `type`) | Description |
| ----------------- | ---------------------------------------------------- |
| `--append` | Append to existing value using key-by-key simulation |
| `--submit` | Press Enter after typing |
Scroll the viewport or scroll an element into view.
```bash theme={null}
brightdata browser scroll # Scroll down 300px (default)
brightdata browser scroll --direction up
brightdata browser scroll --direction down --distance 600
brightdata browser scroll --ref e10 # Scroll element into view
```
| Flag | Description |
| ------------------- | ----------------------------------------------------- |
| `--direction ` | `up`, `down`, `left`, `right` (default: `down`) |
| `--distance ` | Pixels to scroll (default: `300`) |
| `--ref ` | Scroll this element into view instead of the viewport |
Get text or HTML content from the page or a specific element.
```bash theme={null}
# Text content
brightdata browser get text # Full page text
brightdata browser get text "h1" # Text of the first h1
brightdata browser get text "#price" # Text inside #price
# HTML content
brightdata browser get html # Full page outer HTML
brightdata browser get html ".product" # innerHTML of .product
```
Inspect session state.
```bash theme={null}
# HTTP requests captured since last navigation
brightdata browser network
# Cookies for the active session
brightdata browser cookies
# Current session state
brightdata browser status
brightdata browser status --session shop --pretty
# List all active sessions
brightdata browser sessions
```
Navigation controls.
```bash theme={null}
brightdata browser back
brightdata browser forward
brightdata browser reload
```
Close a session and stop its daemon.
```bash theme={null}
brightdata browser close # Close the default session
brightdata browser close --session shop # Close a named session
brightdata browser close --all # Close all active sessions
```
Element `ref` values (e.g. `e1`, `e3`) are re-assigned on every `snapshot` call. After navigating or clicking, take a fresh snapshot before using refs again.
***
## `brightdata zones`
List and inspect Bright Data proxy zones.
```bash theme={null}
brightdata zones # List all active zones
brightdata zones info # Full details for a zone
brightdata zones --json -o zones.json # Export as JSON
brightdata zones info my_zone --pretty # Pretty-print zone info
```
***
## `brightdata budget`
View account balance and per-zone cost/bandwidth. Read-only.
| Subcommand | Description |
| ------------- | -------------------------------------- |
| *(none)* | Quick account balance |
| `balance` | Balance + pending charges |
| `zones` | Cost & bandwidth table for all zones |
| `zone ` | Detailed cost & bandwidth for one zone |
| Flag | Description |
| --------------------- | ------------------------------------------------ |
| `--from ` | Start of date range (e.g. `2024-01-01T00:00:00`) |
| `--to ` | End of date range |
| `--json` / `--pretty` | JSON output |
```bash theme={null}
brightdata budget
brightdata budget balance
brightdata budget zones
brightdata budget zone my_zone
brightdata budget zones --from 2024-01-01T00:00:00 --to 2024-02-01T00:00:00
```
***
## `brightdata config`
View and manage CLI configuration.
| Subcommand | Description |
| ------------------- | ------------------ |
| *(none)* | Show all config |
| `get ` | Get a single value |
| `set ` | Set a value |
| Config Key | Description |
| ----------------------- | ------------------------------------------- |
| `default_zone_unlocker` | Default zone for `scrape` and `search` |
| `default_zone_serp` | Override zone for `search` only |
| `default_format` | Default output format: `markdown` or `json` |
| `api_url` | Override API base URL |
```bash theme={null}
brightdata config
brightdata config set default_zone_unlocker my_zone
brightdata config set default_format json
brightdata config get default_zone_unlocker
```
***
## `brightdata init`
Interactive setup wizard. Walks through authentication, zone selection, and default configuration.
| Flag | Description |
| --------------------- | ---------------------------- |
| `--skip-auth` | Skip the authentication step |
| `-k, --api-key ` | Provide API key directly |
```bash theme={null}
brightdata init
```
***
## `brightdata skill`
Install Bright Data AI agent skills into coding agents (Claude Code, Cursor, Copilot, etc.).
| Subcommand | Description |
| ------------ | -------------------------------------------------- |
| `add` | Interactive picker - choose skills + target agents |
| `add ` | Install a specific skill directly |
| `list` | List all available skills |
Available skills: `search`, `scrape`, `data-feeds`, `bright-data-mcp`, `bright-data-best-practices`
```bash theme={null}
brightdata skill add # Interactive picker
brightdata skill add scrape # Direct install
brightdata skill list # See what's available
```
***
## `brightdata add mcp`
Add the Bright Data MCP server to Claude Code, Cursor, or Codex. Uses the API key stored by `brightdata login`.
```bash theme={null}
brightdata add mcp # Interactive agent + scope prompts
brightdata add mcp --agent claude-code --global
brightdata add mcp --agent claude-code,cursor --project
brightdata add mcp --agent codex --global
```
| Flag | Description |
| ------------------ | --------------------------------------------------------- |
| `--agent ` | Comma-separated targets: `claude-code`, `cursor`, `codex` |
| `--global` | Install to the agent's global config file |
| `--project` | Install to the current project's config file |
### Config file locations
| Agent | Global path | Project path |
| ----------- | --------------------------------------------- | ----------------------- |
| Claude Code | `~/.claude.json` | `.claude/settings.json` |
| Cursor | `~/.cursor/mcp.json` | `.cursor/mcp.json` |
| Codex | `$CODEX_HOME/mcp.json` or `~/.codex/mcp.json` | Not supported |
The command writes the MCP server entry under `mcpServers["bright-data"]`. Existing config is preserved - only the `bright-data` key is added or replaced.
`brightdata add mcp` uses the API key stored by `brightdata login`. It does not read `BRIGHTDATA_API_KEY` or the `--api-key` flag, so run `brightdata login` first.
# Usage examples
Source: https://docs.brightdata.com/cli/examples
Real-world workflows and recipes for the Bright Data CLI (1000+ scrapers), from 1-line scrapes returning clean markdown to scheduled pipelines feeding LLMs.
## How to scrape with the CLI
### Get clean content from any website
```bash theme={null}
# Clean markdown - great for reading or feeding to LLMs
brightdata scrape https://news.ycombinator.com
# Save to file
brightdata scrape https://docs.python.org/3/tutorial/ -o python-tutorial.md
# Get raw HTML for parsing
brightdata scrape https://example.com -f html -o page.html
```
### How to scrape with geo-targeting
```bash theme={null}
# See Amazon prices as a US customer
brightdata scrape https://amazon.com/dp/B09V3KXJPB --country us
# Scrape a German news site from Germany
brightdata scrape https://spiegel.de --country de
# Mobile user agent for mobile-optimized pages
brightdata scrape https://example.com --mobile
```
### How to take screenshots
```bash theme={null}
# Full-page screenshot
brightdata scrape https://example.com -f screenshot -o homepage.png
# Screenshot from a specific country
brightdata scrape https://amazon.co.uk -f screenshot --country gb -o uk-amazon.png
```
***
## How to run search workflows
### Basic web search
```bash theme={null}
# Google search with formatted table output
brightdata search "best web scraping tools 2025"
# Get raw JSON for processing
brightdata search "typescript best practices" --json
# Pretty-print for inspection
brightdata search "AI startups" --pretty
```
### Localized and specialized search
```bash theme={null}
# Local restaurant search from Germany, in German
brightdata search "restaurants berlin" --country de --language de
# News-only results
brightdata search "AI regulation 2025" --type news
# Shopping results
brightdata search "wireless headphones" --type shopping
# Image search
brightdata search "mountain landscape wallpaper" --type images
```
### Pagination
```bash theme={null}
# First page (default)
brightdata search "web scraping tutorials"
# Second page
brightdata search "web scraping tutorials" --page 1
# Third page
brightdata search "web scraping tutorials" --page 2
```
***
## How to run discovery workflows
### How to run AI-powered research
```bash theme={null}
# Basic discovery with formatted table output
brightdata discover "AI trends"
# With AI intent for targeted relevance ranking
brightdata discover "AI trends" \
--intent "Prioritize institutional reports for VC research"
# Include full page content for each result
brightdata discover "AI trends" --include-content --num-results 5
```
### How to discover with geo-targeting
```bash theme={null}
# Local results with date filtering
brightdata discover "best restaurants" --country US --city "New York" \
--start-date 2025-01-01 --end-date 2025-12-31
# Filter results by required keywords
brightdata discover "generative AI SaaS" --filter-keywords "revenue,SaaS"
```
### Export discovery results
```bash theme={null}
# Save as JSON
brightdata discover "AI trends" --num-results 10 --pretty -o results.json
# Pipe-friendly - redirected stdout outputs JSON automatically
brightdata discover "AI trends" --include-content --num-results 3 > results.json
```
***
## How to run browser workflows
### Navigate and read pages
```bash theme={null}
# Open a page and read its content
brightdata browser open https://example.com
brightdata browser snapshot
# Compact snapshot for AI agents (70-90% fewer tokens)
brightdata browser snapshot --compact
# Scope to a specific section
brightdata browser snapshot --selector "main"
```
### Interact with pages
```bash theme={null}
# Open a page
brightdata browser open https://example.com
# Read the page structure
brightdata browser snapshot --compact
# Click, type, and submit using refs from the snapshot
brightdata browser click e3
brightdata browser type e5 "search query" --submit
# Get updated snapshot after interaction
brightdata browser snapshot --compact
# Take a screenshot for visual verification
brightdata browser screenshot ./result.png
# Clean up
brightdata browser close
```
### How to compare multiple sessions
```bash theme={null}
# Open the same page from different countries
brightdata browser open https://amazon.com --session us --country us
brightdata browser open https://amazon.com --session de --country de
# Capture both snapshots
brightdata browser snapshot --session us --json > us.json
brightdata browser snapshot --session de --json > de.json
# Close all sessions
brightdata browser close --all
```
### Extract content from dynamic pages
```bash theme={null}
# Navigate to a page that requires JavaScript
brightdata browser open https://example.com/dashboard
# Get text content from specific elements
brightdata browser get text "#total-revenue"
brightdata browser get text ".summary-table"
# Get HTML for parsing
brightdata browser get html ".product-grid"
# Check network requests
brightdata browser network
```
***
## Structured data extraction
### How to extract e-commerce data
```bash theme={null}
# Amazon product details
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB"
# Amazon product reviews
brightdata pipelines amazon_product_reviews "https://amazon.com/dp/B09V3KXJPB"
# Amazon search - requires keyword + domain
brightdata pipelines amazon_product_search "wireless headphones" "https://amazon.com"
# Walmart product
brightdata pipelines walmart_product "https://walmart.com/ip/123456"
# Export as CSV
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB" --format csv -o product.csv
```
### Social media profiles
```bash theme={null}
# LinkedIn person
brightdata pipelines linkedin_person_profile "https://linkedin.com/in/username"
# LinkedIn company
brightdata pipelines linkedin_company_profile "https://linkedin.com/company/bright-data"
# Instagram profile
brightdata pipelines instagram_profiles "https://instagram.com/username"
# TikTok profile
brightdata pipelines tiktok_profiles "https://tiktok.com/@username"
```
### Reviews and comments
```bash theme={null}
# Google Maps reviews - last 7 days
brightdata pipelines google_maps_reviews "https://maps.google.com/maps/place/..." 7
# YouTube comments - top 50
brightdata pipelines youtube_comments "https://youtube.com/watch?v=dQw4w9WgXcQ" 50
# Facebook company reviews - 25 reviews
brightdata pipelines facebook_company_reviews "https://facebook.com/company" 25
# Instagram comments
brightdata pipelines instagram_comments "https://instagram.com/p/ABC123"
```
***
## Piping and automation
The CLI is designed to be pipe-friendly. When stdout is not a TTY, colors and spinners are automatically disabled.
### Chain search → scrape
```bash theme={null}
# Search Google, extract the first URL, then scrape it
brightdata search "best python frameworks 2025" --json \
| jq -r '.organic[0].link' \
| xargs brightdata scrape
```
### Scrape and read in terminal
```bash theme={null}
# Pipe markdown output to a terminal reader
brightdata scrape https://docs.github.com | glow -
# Or use less
brightdata scrape https://docs.github.com | less
```
### Export to CSV for analysis
```bash theme={null}
# Amazon product data to CSV
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB" --format csv > product.csv
# LinkedIn jobs to CSV
brightdata pipelines linkedin_job_listings "https://linkedin.com/jobs/view/123" --format csv -o jobs.csv
```
### Extract specific fields with jq
```bash theme={null}
# Get just the titles and prices from Amazon search
brightdata pipelines amazon_product_search "laptop" "https://amazon.com" \
| jq '[.[] | {title, price: .final_price}]'
# Get just organic result URLs from search
brightdata search "web scraping" --json | jq -r '.organic[].link'
```
### Async jobs for heavy workloads
```bash theme={null}
# Submit an async scrape
JOB_ID=$(brightdata scrape https://heavy-page.com --async --json | jq -r '.snapshot_id')
# Do other work...
# Check back later
brightdata status $JOB_ID --wait --pretty
```
***
## How to manage your account
### Monitor costs
```bash theme={null}
# Quick balance check
brightdata budget
# Detailed balance with pending charges
brightdata budget balance
# Cost breakdown by zone
brightdata budget zones
# Specific zone in a date range
brightdata budget zone cli_unlocker --from 2024-01-01T00:00:00 --to 2024-02-01T00:00:00
```
### Manage configuration
```bash theme={null}
# View current config
brightdata config
# Set default output to JSON
brightdata config set default_format json
# Use a custom zone for scraping
brightdata config set default_zone_unlocker my_custom_zone
# Override SERP zone
brightdata config set default_zone_serp my_serp_zone
```
***
## AI agent integration
### Add the MCP server to your coding agent
```bash theme={null}
# Interactive - choose agent and scope
brightdata add mcp
# Add to Claude Code globally
brightdata add mcp --agent claude-code --global
# Add to multiple agents at once
brightdata add mcp --agent claude-code,cursor --project
# Add to Codex
brightdata add mcp --agent codex --global
```
### Install skills into coding agents
```bash theme={null}
# Interactive picker - choose skills and target agents
brightdata skill add
# Install the scraping skill into your agent
brightdata skill add scrape
# Install search capabilities
brightdata skill add search
# See all available skills
brightdata skill list
```
Skills are pre-packaged bundles of prompts and configuration that teach AI coding agents how to use Bright Data effectively. See [Skills](/ai/for-agents/skills) for more details.
***
## Supported environment variables
Override any stored configuration with environment variables:
| Variable | Purpose |
| ---------------------------- | ------------------------------------------------------ |
| `BRIGHTDATA_API_KEY` | API key (skips login entirely) |
| `BRIGHTDATA_UNLOCKER_ZONE` | Default Web Unlocker zone |
| `BRIGHTDATA_SERP_ZONE` | Default SERP zone |
| `BRIGHTDATA_POLLING_TIMEOUT` | Polling timeout in seconds |
| `BRIGHTDATA_BROWSER_ZONE` | Default Scraping Browser zone (default: `cli_browser`) |
```bash theme={null}
# Use in CI/CD without login
BRIGHTDATA_API_KEY=your_key brightdata scrape https://example.com
# Override timeout for large pipeline jobs
BRIGHTDATA_POLLING_TIMEOUT=1200 brightdata pipelines amazon_product "https://amazon.com/dp/..."
```
# Bright Data CLI FAQs
Source: https://docs.brightdata.com/cli/faqs
FAQs and troubleshooting for the Bright Data CLI: brightdata and bdata aliases, authentication, installation issues and common errors across 1000+ scrapers.
## Frequently asked general questions
They are identical - `bdata` is a shorthand alias installed alongside `brightdata`. Use whichever you prefer.
Yes. The CLI connects to Bright Data's infrastructure to handle web requests. You can [sign up for free](https://brightdata.com/?hs_signup=1\&utm_source=docs) and get started with the free tier.
The CLI supports 40+ platforms through the `pipelines` command, including Amazon, LinkedIn, Instagram, TikTok, YouTube, Facebook, Reddit, Google Maps, Walmart, eBay, and many more. Run `brightdata pipelines list` to see the full list.
* **Scrape:** `markdown` (default), `html`, `json`, `screenshot`
* **Search:** Formatted table (default), `json`, `pretty`
* **Pipelines:** `json` (default), `csv`, `ndjson`, `jsonl`
All commands support `-o ` to write output to a file.
Yes. The CLI is fully pipe-friendly. When stdout is not a TTY, colors and spinners are automatically disabled. Use `--json` for machine-readable output and `BRIGHTDATA_API_KEY` environment variable for non-interactive authentication.
## Authentication
Credentials are stored locally on your machine:
| OS | Path |
| ------- | --------------------------------------------------------------- |
| macOS | `~/Library/Application Support/brightdata-cli/credentials.json` |
| Linux | `~/.config/brightdata-cli/credentials.json` |
| Windows | `%APPDATA%\brightdata-cli\credentials.json` |
The file is set to mode `0o600` (owner read/write only).
Use the device flow:
```bash theme={null}
brightdata login --device
```
This prints a URL and a verification code. Open the URL on any device with a browser, enter the code, and authentication completes on the server.
Run `brightdata logout` followed by `brightdata login` with the new account. Or pass a different API key directly:
```bash theme={null}
brightdata login --api-key
```
## Troubleshooting
This means no default zone is configured. Fix it by either:
```bash theme={null}
# Re-run login (auto-creates zones)
brightdata login
# Or set a zone manually
brightdata config set default_zone_unlocker
```
Your stored API key is no longer valid. Re-authenticate:
```bash theme={null}
brightdata login
```
Your API key does not have permission for the requested zone or operation. Check zone permissions in the [Bright Data control panel](https://brightdata.com/cp).
You've hit the rate limit for your zone. Options:
* Wait a moment and retry
* Use `--async` for large jobs to avoid blocking
* Contact your account manager to increase limits
The default polling timeout is 600 seconds (10 minutes). For large datasets, increase it:
```bash theme={null}
# Per-command
brightdata pipelines amazon_product "" --timeout 1200
# Via environment variable
export BRIGHTDATA_POLLING_TIMEOUT=1200
```
The CLI auto-detects TTY support. If detection fails, pipe through `cat` to force plain output:
```bash theme={null}
brightdata scrape https://example.com | cat
```
Or use `--json` for clean, parseable output.
## Frequently asked configuration questions
```bash theme={null}
brightdata config set default_format json
```
Valid values: `markdown`, `json`.
Settings are resolved in this order (highest priority first):
1. **CLI flags** - e.g., `--zone my_zone`
2. **Environment variables** - e.g., `BRIGHTDATA_UNLOCKER_ZONE`
3. **config.json** - e.g., `default_zone_unlocker`
4. **Defaults** - built-in fallbacks
Delete the configuration directory:
```bash theme={null}
# Linux
rm -rf ~/.config/brightdata-cli/
# macOS
rm -rf ~/Library/Application\ Support/brightdata-cli/
```
Then run `brightdata login` to start fresh.
# Installation & setup
Source: https://docs.brightdata.com/cli/installation
Install the Bright Data CLI (1000+ scrapers) via npm or curl and authenticate your account with an API key in 1 minute. Includes setup verification commands.
## Install the CLI
The fastest way to run the Bright Data CLI is `npx`, which executes the latest version with no global install:
```bash theme={null}
npx -p @brightdata/cli brightdata --version
```
Prefix any command in these docs with `npx -p @brightdata/cli` to run it the same way, with no global dependency to maintain. The Bright Data CLI requires Node.js 20 or later.
For a persistent `brightdata` command and faster startup, install the CLI globally instead:
```bash npm (recommended) theme={null}
npm install -g @brightdata/cli
```
```bash yarn theme={null}
yarn global add @brightdata/cli
```
```bash pnpm theme={null}
pnpm add -g @brightdata/cli
```
Verify a global install:
```bash theme={null}
brightdata --version
```
The shorthand alias `bdata` is also available, both with `npx -p @brightdata/cli bdata` and after a global install - use whichever you prefer.
## Update the CLI
Upgrade to the latest version with the same package manager you installed with:
```bash npm theme={null}
npm install -g @brightdata/cli@latest
```
```bash yarn theme={null}
yarn global add @brightdata/cli@latest
```
```bash pnpm theme={null}
pnpm add -g @brightdata/cli@latest
```
Check your installed version against the latest published version:
```bash theme={null}
brightdata --version # your installed version
npm view @brightdata/cli version # latest version on npm
```
New commands ship in CLI releases. For example, `scraper heal` and `scraper approve` were added in v0.3.1. Run the update command above to pick them up. See the [release notes](https://github.com/brightdata/cli/releases) for what changed in each version.
## Authenticate
Run the login command to connect your Bright Data account:
```bash theme={null}
brightdata login
```
This opens your browser for secure OAuth authentication. Once complete, the CLI:
1. Validates and stores your API key locally
2. Auto-creates required proxy zones (`cli_unlocker`, `cli_browser`)
3. Sets sensible defaults so you can start immediately
You only need to log in **once**. All subsequent commands authenticate automatically.
### Alternative authentication methods
When no browser is available, use the device flow:
```bash theme={null}
brightdata login --device
```
This prints a URL and a code. Open the URL on any device, enter the code, and the CLI completes authentication.
For CI/CD pipelines or non-interactive environments, pass your API key directly:
```bash theme={null}
brightdata login --api-key YOUR_API_KEY
```
You can find your API key in the [Bright Data control panel](https://brightdata.com/cp/setting).
Set the `BRIGHTDATA_API_KEY` environment variable to skip login entirely:
```bash theme={null}
export BRIGHTDATA_API_KEY=YOUR_API_KEY
```
This is useful for Docker containers, GitHub Actions, and other automated environments.
## Interactive setup wizard
For a guided first-time experience, use the init command:
```bash theme={null}
brightdata init
```
This walks you through authentication, zone selection, and default configuration step by step.
| Flag | Description |
| --------------------- | --------------------------------------------------- |
| `--skip-auth` | Skip the authentication step (if already logged in) |
| `-k, --api-key ` | Provide API key directly |
## Verify your setup
After logging in, confirm everything is working:
```bash theme={null}
# Check your configuration
brightdata config
# Verify API connectivity
brightdata budget
# Try a quick scrape
brightdata scrape https://example.com
```
## Where configuration is stored
The CLI stores credentials and configuration locally:
| OS | Path |
| ------- | ----------------------------------------------- |
| macOS | `~/Library/Application Support/brightdata-cli/` |
| Linux | `~/.config/brightdata-cli/` |
| Windows | `%APPDATA%\brightdata-cli\` |
Two files are created:
| File | Purpose | Permissions |
| ------------------ | --------------------------------- | -------------------- |
| `credentials.json` | API key | `0o600` (owner-only) |
| `config.json` | Zones, output format, preferences | Standard |
**Priority order for configuration:** CLI flags → Environment variables → `config.json` → Defaults. You can always override any setting on a per-command basis.
## Next steps
Explore the full command reference.
Jump into real-world workflows and recipes.
# Bright Data CLI
Source: https://docs.brightdata.com/cli/overview
Scrape websites, search the web, extract structured data from 40+ platforms, and manage your Bright Data account from the terminal.
Install the CLI, log in once, and start scraping - CAPTCHAs, anti-bot protections, and JavaScript rendering are handled automatically.
## What can you do with the CLI?
Get clean markdown, HTML, JSON, or screenshots from any URL - anti-bot bypass and JS rendering included.
Query Google, Bing, or Yandex and get structured results with organic listings, ads, and People Also Ask.
AI-powered web discovery - find, rank, and extract content by intent with relevance scoring.
Pull product details, profiles, reviews, and more from 40+ platforms like Amazon, LinkedIn, Instagram, and TikTok.
Navigate, click, type, screenshot, and read pages using a real remote browser session with persistent state.
Install skills or add the Bright Data MCP server into Claude Code, Cursor, Codex, and other AI coding agents.
List proxy zones, inspect configurations, and monitor account balance and per-zone costs.
JSON output, file export, and pipe-friendly design make it easy to chain commands and build workflows.
## What the CLI looks like
```bash theme={null}
# Scrape a page as clean markdown
brightdata scrape https://news.ycombinator.com
# Search Google and get structured results
brightdata search "best web scraping tools 2025"
# AI-powered discovery with relevance ranking
brightdata discover "AI trends" --intent "Prioritize institutional reports for VC research"
# Extract an Amazon product as JSON
brightdata pipelines amazon_product "https://amazon.com/dp/B09V3KXJPB" --format json
# Control a remote browser session
brightdata browser open https://example.com
brightdata browser snapshot --compact
# Add Bright Data MCP server to your coding agent
brightdata add mcp
```
The CLI alias `bdata` is available as a shorthand - e.g., `bdata scrape https://example.com`.
## How it works
The Bright Data CLI wraps the full Bright Data web data platform into simple terminal commands. Under the hood, it:
1. **Authenticates once** via OAuth, device flow, or API key - credentials are stored locally and never need to be entered again
2. **Auto-provisions zones** (`cli_unlocker`, `cli_browser`) on first login so you can start immediately
3. **Routes requests** through Bright Data's infrastructure, handling CAPTCHAs, bot detection, IP rotation, and JavaScript rendering
4. **Returns clean output** - formatted tables in the terminal, or structured JSON/CSV/markdown for automation
## Explore CLI resources
Install the CLI and authenticate with Bright Data.
Full reference for every command, flag, and option.
Real-world workflows and recipes for common tasks.
Answers to common questions and troubleshooting tips.
**Zero configuration required.** After a one-time `brightdata login`, every command works out of the box - no tokens to manage, no zones to create, no proxies to configure.
# Archive API reference
Source: https://docs.brightdata.com/datasets/archive/api-reference
API reference for Bright Data's Web Archive search, including endpoints for running queries and retrieving archived web data.
# How data range affects delivery time
Source: https://docs.brightdata.com/datasets/archive/data-range-vs-delivery-time
Understand how your selected data range affects delivery time in Bright Data's Web Archive, including S3 Glacier retrieval delays for older data.
If your query is matching data within **last 24 hours** - your snapshot will start processing/delivering immediately.
If some of your matched data is **older than 24 hours** - it needs to be retrieved from **S3 Glacier Deep Archive** storage tier before delivery, which may take **up to 72 hours**.
Avoid queries that span the retention boundary (approximately 24 hours from now).
Requests with `max_age` or time ranges that fall within \~24h ± 2h of the current time may include files that have already been migrated to archive storage tier. Attempting a dump for such queries can cause the dump to stall or remain incomplete because of files storage class transition.
**Recommendations:**
* We recommend using `max_age` = `24h` for initial testing to ensure fast delivery.
* For real-time data needs: use `max_age: "24h"` or a narrower window to avoid the retention edge.
* For historical data (older than 24h): use explicit `min_date`/`max_date` filters rather than `max_age`.
* If a dump appears stalled: we usually retry automatically, please open a ticket if it didn't happen.
# FAQ: Archive API
Source: https://docs.brightdata.com/datasets/archive/faqs
FAQs about the Bright Data Archive API: available data, freshness, refresh cadence, pricing model and 4 supported delivery destinations for archive snapshots.
Archive API is a massive, continuously expanding, cached repository by Bright Data, designed to capture and deliver public web data at scale.
It provides full web pages and metadata, making it ideal for AI training, machine learning, and large-scale data analysis.
Unlike traditional web crawls, Archive API prioritizes **relevance**, **freshness**, and **usability**, giving you access to the most important parts of the internet as they are scraped daily.
Bright Data’s Archive API has already collected \~90 PB of data, covering \~500 billion+ unique URLs from \~380 million domains.
We continue to add \~1 PB of new data every week, alongside \~2 billion unique URLs, making Archive the **largest**, up-to-date, web data repository available - perfect for AI and data-driven applications.
You can start accessing data immediately through our [Archive API](/datasets/archive/overview). The API allows you to search, retrieve and filter data snapshots from Archive seamlessly and efficiently.
* Data from the last 24 hours: Will take from within minutes and up to a few hours to deliver (depending on snapshot size)
* Data older than 24 hours: Will take up to 72 hours to process and deliver (depending on snapshot size)
Archive offers three delivery options to ensure seamless integration into your existing workflows:
* **Amazon S3 bucket:** Have your Data Snapshot delivered directly to your S3 bucket.
* **Azure Blob Storage:** Deliver the data snapshot directly to your Azure Blob container.
* **Webhook:** Retrieved via webhook for real-time integration into your systems.
Absolutely! Archive API allows filtering by category, domains, date, languages, and country before retrieving data, ensuring you only get what you need.
When working with large-scale web data, **freshness**, **relevance**, and **accessibility** are key. While Common Crawl provides a broad historical snapshot of the web, Bright Data’s Archive API offers real-time, continuously updated data with advanced filtering and delivery options. Here’s how they compare:
| **Feature** | **Bright Data’s Archive** | **Common Crawl** |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| **Data Collection** | Continuously captures public web data in real time, providing results as recent as “now.” | Periodic web crawling (not real-time), updated monthly or bimonthly. Data can be outdated |
| **Data Volume** | \~90 PB collected in a few years, covering \~500 billion pages across \~380M domains. Adds \~1 PB and \~2 billion unique URLs/week. | 250b pages collected over 18 years. |
| **Website Coverage & Relevance** | Focuses on high-value, relevant website data based on real scraping business needs. | Crawls indiscriminately, including outdated or low-quality pages. |
| **Data Types** | Full web pages (JS-rendered) | 98.6% HTML and text |
| **Filtering & Delivery** | Full discovery and delivery platform- filtering by category, domain, language, date etc. Delivered via Amazon S3, Azure or webhook. | No built-in filtering or delivery. Need to manually process huge raw WARC files. |
# Web Archive overview
Source: https://docs.brightdata.com/datasets/archive/overview
The Web Archive gives access to Bright Data's stored web traffic (250+ domains), a growing repository of pages collected through Unlocker and SERP APIs.
## What it does
Instead of running your own crawlers, you search the archive, filter what you need (by time range, domain, URL patterns, language, blocking signals), and export ready-to-use datasets as HTML files + metadata.
## Common use cases
* **LLM training and RAG pipelines**: Build or refresh training corpora from targeted web segments
* **Search and indexing**: Backfill indexes with historical content across large domain sets
* **Search product augmentation**: Improve coverage for sites with advanced blocking, supporting reliable page retrieval at scale
## How it works
Filter by time range, domains, URL patterns, language, or signals (CAPTCHA, robots blocks, etc.)
See matched file count, snapshot size, expected duration, and cost
Export the snapshot as HTML files + metadata (URL, timestamp, collection attributes) to Amazon S3, Azure Blob Storage, or via webhook
# Data validation for customers
Source: https://docs.brightdata.com/datasets/data-validation/data-validation-for-customers
How Bright Data's automated dataset validation works across 250+ domains for customers: verification checks, error handling and approval flow before delivery.
## Overview
Bright Data’s automated dataset creation platform consists of a verification and approval phase before the dataset delivery. Bright Data facilitates error handling, validation checks, and customization, ensuring data accuracy and reliability. These validation checks are crucial in saving time, reducing errors in the data, and keeping the data quality at the desired level.
## How does it work?
Once the dataset snapshot is ready:
The user will get the dataset with an indication in the Bright Data Control Panel that all tests passed.
The developer reviews the issues and will decide whether to:
1. Fix the dataset according to the failed tests.
2. Deliver the dataset to the user, explaining why the validation test failed but was overridden.
The user can then decide to:
1. Approve the snapshot.
2. Approve the snapshot for this time frame only.
3. Reject the snapshot, and we will fix the scraper accordingly.
When the user approves the dataset snapshot, they proceed to the delivery phase.
## Which validation rules apply
The dataset must contain a certain percentage of unique values.
* Example: In the LinkedIn company profiles dataset, each company's LinkedIn URL should be unique. If duplicate URLs exist, the same company is listed more than once, violating the uniqueness rule.
The dataset must contain a minimum percentage of filled values.
* Example: In a LinkedIn company profiles dataset, at least 90% of the profiles must have the 'Industry' field filled. If more than 10% of the profiles are missing this information (leaving the 'Industry' field blank), the dataset does not meet the required filling rate.
Certain fields must be filled; an error will occur if they remain empty.
* Example: Fields like 'Company Name' and 'Headquarters Location' might be mandatory in the LinkedIn dataset. Any profile lacking this information is flagged as an error.
Compared to previous values collected, the number value must not change by more than X.
* Example: If the dataset is updated regularly, the number of employees for a company shouldn’t drastically change between updates (e.g., a sudden jump from 50 to 5000 employees) unless there's a known reason (like a merger).
Verifies each entry's data type against its field type (e.g., string, number, date) and flags mismatches for correction.
* Example: The dataset should only accept date formats in the 'Foundation Date' field. If a text string like 'unknown' is entered, it should be flagged for correction.
Create a custom rule to validate whether a field exists and its value is valid, such as requiring the size string to be 'S,' 'M,' or 'L.'
* Example: The dataset might have a field for 'Company Size' with accepted values like 'Small,' 'Medium,' and 'Large.' A record must be flagged if it has a value outside these options.
The dataset must have X records (each URL should have X records from the total URL inputs).
* Example: If the dataset aims to represent companies in a specific sector, like technology, it must have a minimum number of company profiles from this sector to be considered complete and representative.
Determines whether dataset size fluctuations are within a +/- X% range.
* Example: For datasets updated monthly, the total number of companies listed shouldn’t fluctuate wildly (e.g., more than 10% increase or decrease) from one month to the next unless a specific event or trend is affecting the industry.
Checks each individual record in the dataset to ensure it doesn't have a high percentage of empty or null fields. If a record has more than a predetermined threshold (e.g., 70%) of its fields empty or null, it triggers an error.
* Example: In the LinkedIn company profiles dataset, if a specific company's profile has more than 70% of its fields (like industry, size, location, description) empty, this rule would flag it as incomplete.
Detects and resolves issues that lead to multiple duplicate records in the dataset due to improper identity assignment or entry errors. It ensures each record is distinct and accurately represents a unique data point.
* Example: In the LinkedIn dataset, this rule would identify cases where the same company is listed multiple times due to errors in assigning unique identifiers. For instance, if slight variations in company profiles (like different spellings of a company name) result in the same company appearing as separate entries, this rule would flag them for correction.
## Main components and functionality
### Evaluating the validation test results
Once the dataset snapshot validation errors are handled, the user is notified to evaluate and choose whether to:
1. Approve
2. Approve temporarily
3. Reject the snapshot.
### How to act on evaluation results
For each failed validation test, the user has three options:
1. **Set a new threshold**
1. Set custom values - If the developer doesn't reach the default set value, the user can choose a new threshold. The snapshot is returned to the developer once a new threshold is set.
2. Set for X% - Accept the success rate the developer reached and set the threshold to the value the developer managed to extract.
2. **Ignore test (one time only)** - Accept the value that the developer extracted just once (the default value will not change for the next dataset snapshot)
3. **Reject** - The user doesn’t accept the adjustments to the failed tests; the issues will be returned to the developer to fix. The status will be marked as “Rejected” for additional fixes and will later on be re-send to the customer for approval.
In case all issues are ignored/approved, click on “Deliver dataset” to deliver the snapshot.
A snapshot delivery will be automatically delivered when pending customer approval for 14 days.
In case all/some issues are rejected, click on “send back to the developer” to send it back for additional fixes.
## Communications and notifications
Users are notified of status updates through their account in the control panel and via email.
# Data validation for partners
Source: https://docs.brightdata.com/datasets/data-validation/data-validation-for-partners
How Bright Data's data staging and validation process works across 250+ domains for partners: error handling, approval workflows and quality checks.
## Overview
We are introducing a new data staging process, enabling both developers and owners to verify and approve datasets before delivery.
The system will facilitate error handling, validation checks, and customization, ensuring accuracy and reliability.
It will help everyone save time, reduce the number of open tickets, and keep the needed level of quality to maintain the level of quality we want.
## How does it work?
Once the dataset snapshot is ready, we run its validation tests.
The customer will get the dataset snapshot with an indication on the CP that all test was passed.
The partner will review the issues and will choose:
1. To fix the failed tests
2. To deliver the snapshot as it is to the customer (**with an explanation of why this test failed but still override**).
The customer can then decide (per fail test or as a batch):
1. If he is willing to accept it as it is
2. Accept it as it is only for this specific snapshot
3. Reject it and return it to the partner to fix what needs to be fixed
Once the customer approves, the snapshot goes to the delivery phase.
## The validation tests
Ensures a minimum percentage of unique values in the dataset.
Mandates a minimum percentage for filled values.
Makes a field mandatory once filled; triggers an error if left empty afterward.
The value number must not change by more than X amount compared to previous values.
Checks each entry's data type against its field type (e.g., string, number, date) to ensure integrity and flag mismatches for correction before processing.
Establish a custom rule to validate if the specific field exists and the field value is valid, such as requiring the size string to be 'S', 'M', or 'L'; any other value is considered an error.
Requires a minimum of X records for the initial dataset (in specific URL should per the minimum of X records from the total URL inputs)
Validates fluctuations within a +/- X% range.
## **Main components and functionality**
There are three filter options (All results, Passed, Failed) for dataset test view
This will allow you to reparse the cached data
An option to override a test because the data seems healthy/or it makes sense that the specific dataset does not meet the threshold (note! In case of an override, you will need to write an explanation to the customer)
In case the test failed, click on the “show related records” to see example of the records and then on “View\&Edit code” to get to the IDE and start fixing the issues
Reparse:
Reparse involves reprocessing the existing raw data that has already been collected.
It applies the parsing logic, rules, or transformations to the data without fetching fresh data from the web.
Reparse is useful when changes are needed in data structure, new fields are added to parsing logic, or existing data needs to be restructured or re-extracted.
Recrawl:
Recrawl involves revisiting the source website to collect new or updated data.
It fetches fresh data directly from the source to reflect recent changes or additions on the website.
Recrawling is useful when the content on the source website changes frequently, or when up-to-date data is critical for analysis or reporting.
Once you finish working on the needed fix, you’ll be able to Reparse or recrawl according to your needs
This will allow you to run the validation tests again if needed
This will allow you to redirect to the IDE in context in case you need to edit the collector and recrawl (e.g in case there are no records at all)
After reviewing the needed data and fixing/ignoring issues, you should click on this button and send the snapshot to the owner for review. The ignored tests should be equipped with a reasoning to explain why you chose to ignore the test
In case not all rejected test was accepted by the owner, the issues will be sent back to you and will be marked with the “Rejected” label for additional fixes and re-sending to the owner for approval
## Communications and notifications
Status changes and additional notifications will be presented to you as a megaphone on the CP.
## Tickets and bugs
Now, once we introduce the staging process, fixing a collector is not the end of the bug/issue
The process consists of two steps:
1. Fix the collector
2. Fix the snapshot
Hence, the bug processing workflow will be amended to align with the new 2-step process.
Tickets related to validation issues should not be allowed to be marked as “resolved” before the snapshot is delivered to the dataset owner!
## Changes to the flow:
We are removing the “Resolve” button from the option on the ticket
# Code examples
Source: https://docs.brightdata.com/datasets/deep-lookup/code-examples
Python, Node.js and shell examples for the Bright Data Deep Lookup API: submit a query, poll for results and parse the structured response end-to-end.
```python Python - Complete Research Flow theme={null}
import requests
import time
class DeepLookupAPI:
def __init__(self, api_key):
self.api_key = api_key
self.base_url = "https://api.brightdata.com/datasets/deep_lookup/v1"
self.headers = {"Authorization": f"Bearer {api_key}"}
def research_with_spec(self, query, columns, limit=100):
# Create detailed specification
spec = {
"name": "companies",
"query": query,
"title": query.replace("Find all ", ""),
"columns": columns
}
# Trigger research
trigger_response = requests.post(
f"{self.base_url}/trigger",
headers=self.headers,
json={
"query": query,
"spec": spec,
"result_limit": limit
}
).json()
request_id = trigger_response["request_id"]
# Poll for completion
while True:
status_response = requests.get(
f"{self.base_url}/request/{request_id}/status",
headers=self.headers
).json()
print(f"Progress: {status_response.get('progress', 0)}%")
if status_response["status"] == "completed":
break
elif status_response["status"] == "failed":
raise Exception("Research failed")
time.sleep(5)
# Get results
results = requests.get(
f"{self.base_url}/request/{request_id}",
headers=self.headers
).json()
return results
def monitor_progress(self, request_id):
"""Monitor detailed progress of a research request"""
while True:
result = requests.get(
f"{self.base_url}/request/{request_id}",
headers=self.headers
).json()
step = result.get('step', 'unknown')
if step == 'identifying':
print("Analyzing your query...")
elif step == 'generating_schema':
print("Creating data structure...")
elif step == 'generating':
pages = result.get('pages_read', 0)
matched = result.get('matched_records', 0)
print(f"Processing data: {pages} pages read, {matched} matches found")
elif step == 'done':
print("Research completed!")
return result
time.sleep(3)
# Usage
api = DeepLookupAPI("YOUR_API_KEY")
columns = [
{
"name": "company_name",
"description": "Name of the company",
"type": "enrichment"
},
{
"name": "is_ai_company",
"description": "Must be an AI/ML focused company",
"type": "constraint"
},
{
"name": "employee_count",
"description": "Number of employees",
"type": "enrichment"
},
{
"name": "min_50_employees",
"description": "Must have at least 50 employees",
"type": "constraint"
}
]
results = api.research_with_spec(
"Find all AI companies in Israel with more than 50 employees",
columns,
limit=100
)
print(f"Found {results['matched_records']} companies")
print(f"Skipped {results['skipped_records']} companies (didn't match all criteria)")
print(f"Total cost: {results['total_cost']}")
```
```javascript Node.js - Preview and Execute with Progress Monitoring theme={null}
const axios = require('axios');
class DeepLookupAPI {
constructor(apiKey) {
this.apiKey = apiKey;
this.baseURL = 'https://api.brightdata.com/datasets/deep_lookup/v1';
this.headers = {
'Authorization': `Bearer ${apiKey}`,
'Content-Type': 'application/json'
};
}
async previewAndExecute(query, limit = 100) {
// Create preview
const previewResponse = await axios.post(
`${this.baseURL}/preview`,
{ query },
{ headers: this.headers }
);
const previewId = previewResponse.data.preview_id;
// Wait for preview to complete
let previewData;
do {
await new Promise(resolve => setTimeout(resolve, 2000));
const response = await axios.get(
`${this.baseURL}/preview/${previewId}`,
{ headers: this.headers }
);
previewData = response.data;
} while (previewData.status !== 'completed');
console.log('Preview ready with', previewData.sample_data.length, 'samples');
// Trigger full research
const triggerResponse = await axios.post(
`${this.baseURL}/trigger`,
{
preview_id: previewId,
result_limit: limit
},
{ headers: this.headers }
);
const requestId = triggerResponse.data.request_id;
// Monitor detailed progress
let lastStep = '';
let result;
do {
await new Promise(resolve => setTimeout(resolve, 3000));
const response = await axios.get(
`${this.baseURL}/request/${requestId}`,
{ headers: this.headers }
);
result = response.data;
if (result.step !== lastStep) {
lastStep = result.step;
switch(result.step) {
case 'identifying':
console.log('Analyzing query...');
break;
case 'generating_schema':
console.log('Creating data structure...');
break;
case 'generating':
console.log('Collecting data from sources...');
break;
case 'done':
console.log('Research completed!');
break;
}
}
if (result.step === 'generating' && result.matched_records) {
console.log(` Found ${result.matched_records} matches so far...`);
}
} while (result.step !== 'done' && result.status !== 'failed');
return result;
}
async enrichResults(requestId, columnName, columnQuery) {
// Add enrichment column
const enrichResponse = await axios.post(
`${this.baseURL}/request/${requestId}/enrich`,
{
column_name: columnName,
query: columnQuery
},
{ headers: this.headers }
);
console.log(`Adding "${columnName}" column...`);
console.log(`Maximum additional cost: ${enrichResponse.data.max_additional_cost}`);
// Wait for enrichment to complete
// (Implementation would depend on actual API behavior)
return enrichResponse.data;
}
}
// Usage
const api = new DeepLookupAPI('YOUR_API_KEY');
async function runResearch() {
const results = await api.previewAndExecute(
'Find all B2B marketplaces in Europe',
50
);
console.log(`Found ${results.matched_records} marketplaces`);
console.log(`Skipped ${results.skipped_records} that didn't match criteria`);
console.log(`Cost: ${results.total_cost}`);
// Add an enrichment column
await api.enrichResults(
results.request_id,
'ceo_name',
'CEO or founder name'
);
}
runResearch();
```
# Error handling
Source: https://docs.brightdata.com/datasets/deep-lookup/errors
Reference for Deep Lookup API error codes and response formats, including INVALID_QUERY and other common errors with resolution guidance.
## Error Response Format
```json theme={null}
{
"error": {
"code": "INVALID_QUERY",
"message": "Query must start with 'Find all'",
"details": "Please rephrase your query to begin with 'Find all' followed by what you're looking for"
}
}
```
## Common Error Codes
| Code | Description | Resolution |
| :--------------------- | :------------------------------- | :-------------------------- |
| `INVALID_API_KEY` | API key is missing or invalid | Check your API key |
| `RATE_LIMIT_EXCEEDED` | Too many requests | Wait and retry with backoff |
| `INVALID_QUERY` | Query format is incorrect | Follow query guidelines |
| `INSUFFICIENT_CREDITS` | Account has insufficient credits | Add credits to account |
| `REQUEST_NOT_FOUND` | Request ID doesn't exist | Verify request ID |
| `PROCESSING_ERROR` | Internal processing error | Contact support |
# Deep Lookup documentation
Source: https://docs.brightdata.com/datasets/deep-lookup/overview
Deep Lookup is an AI-powered research tool for finding companies, professionals, and entities from 1,000+ public sources with 95%+ accuracy.
## Introduction
Deep Lookup is an AI-powered research tool that allows you to search the public web like a database. Find any set of companies or professionals with precision and get table-ready structured data. With Deep Lookup, you can quickly identify real-world entities such as professionals, companies, products, news and articles, locations, and events that meet specific filters and criteria.
## Why Deep Lookup?
Turn weeks of manual research into minutes. Deep Lookup searches 1,000+ sources simultaneously, delivering comprehensive results instantly.
Our advanced AI models validate data across multiple sources, ensuring the highest quality results for critical business decisions.
No complex query builders or SQL knowledge needed. Simply describe what you're looking for in plain English, starting with "Find all..."
Never pay for unmatched results. Our transparent pricing means you only pay for the verified data you receive, not for skipped or filtered-out records.
## Key Features
**Universal Entity Search**\
Find companies, professionals, products, news, locations, and events using specific, measurable criteria.
**Smart Data Enrichment**\
Automatically add valuable data points to your results:
* Contact information and emails
* Revenue and employee counts
* Technology stacks and funding data
* Social profiles and company details
**Preview Mode**\
Get 10 free sample records to validate your approach before running the full research.
**Post-Run Enhancement**\
Add new data columns even after your initial research is complete.
**Source Transparency**\
See exactly where each data point comes from with full source attribution.
**Real-Time Extraction**\
Access the most current data available from the public web, not outdated databases.
## How Deep Lookup Works
Begin with "Find all" followed by what you're looking for.
The more specific your query, the better your results. Deep Lookup works best when you provide clear, measurable criteria.
Get 10 free sample records to validate your approach before running the full research.
## Understanding Column Types
When building your research in Preview Mode, you'll work with two types of columns:
### How enrich columns work
**Purpose:** Add new attributes to each result using data from the public web.
These columns provide additional information about your entities without filtering results. For example:
* CEO name for a company
* Revenue figures
* Contact information
* Social media profiles
* Technology stack used
### How filter columns work
**Purpose:** Limit results by applying specific conditions to your query.
These columns determine which records are included in your dataset. For example:
* Companies with revenue greater than \$10 million
* Professionals with 5+ years experience
* Products priced under \$100
* Articles published in the last 30 days
## Understanding "Skipped" Results
When you see "skipped" in a cell, it means the candidate entity did not match at least one of your filter criteria. This is how Deep Lookup ensures precision in your results.
### Important Points About Skipped Results:
* **More filters = fewer results**: The more filters you apply, the more precise your results become, but you may get fewer total matches
* **Each filter must be satisfied**: All filter conditions must be met for an entity to be included
* **Optimization tip**: If you're getting too many skipped results, consider removing some filters to broaden your search
* **Quality over quantity**: Skipped results ensure you only pay for data that exactly matches your criteria
### Example:
If you search for "Find all SaaS companies in California with >100 employees AND revenue >\$10M AND founded after 2020":
* A company with 150 employees and \$15M revenue but founded in 2019 would be skipped
* Only companies meeting ALL criteria appear in your final results
## Deep Lookup Modes
### Preview Mode (Recommended)
The best way to ensure accurate results before spending credits:
* **Free preview results** - Get up to 10 sample records at no cost to understand expected outcomes
* **Query refinement** - Deep Lookup's AI assistant helps you refine and improve query accuracy
* **Column customization** - Review and adjust column names, types, and specifications before running
* **Specs panel control** - Tweak column settings, reorder fields, change types, or add new columns
* **Runtime estimates** - See processing time and cost estimates before committing
### When to use Instant Mode
For fast, one-shot results when you know exactly what you need:
* Run queries immediately for simple, well-defined searches
* Best for repeat queries with proven parameters
* Skip the preview step when you're confident in your criteria
### When to use Advanced Mode
For complex, multi-step research:
* Build sophisticated queries with multiple refinement steps
* Perfect for deep market research or complex data gathering
* Combine multiple data sources and relationships
## Writing Effective Queries
### The Basic Structure
Start every query with "Find all" and describe exactly what you need:
```text theme={null}
Find all B2B SaaS companies in Texas with revenue greater
than $10 million and less than 100 employees founded after 2020
```
### Key Success Tips
**1. Use Natural Language Comparisons**
* "greater than 50 employees"
* "less than \$5 million in revenue"
* "between 100 and 500 employees"
**2. Specify Geographic Boundaries**
* "in California"
* "headquartered in London"
* "operating in Southeast Asia"
**3. Add Revenue or Size Constraints**
* "with annual revenue greater than \$20 million"
* "having between 50 and 200 employees"
**4. Combine 2-4 Specific Criteria**
The sweet spot for queries is combining a few specific requirements
**5. Include Industry-Specific Terms**
* For tech: "SaaS", "API-first", "cloud-native"
* For finance: "Series B funded", "EBITDA positive"
* For retail: "D2C", "omnichannel", "subscription-based"
## Popular Use Cases
### Sales & Lead Generation
**B2B Decision Maker Targeting** - 100% match rate
```text theme={null}
Find all VPs of Sales at fintech companies in New York with
50-200 employees including their verified email addresses
```
### How to research competitors
**Competitor Pricing Research** - 97% match rate
```text theme={null}
Find all project management software with pricing between
$10-100 per user that offer kanban board features
```
### Recruitment & Talent Acquisition
**Executive Search** - 96% match rate
```text theme={null}
Find all Chief Technology Officers at healthcare companies
in California with more than 500 employees
```
### How to run market research
**Industry Analysis** - 95% match rate
```text theme={null}
Find all electric vehicle manufacturers in the United States
with manufacturing facilities and more than 500 employees
```
### How to research investments
**Deal Flow Discovery** - 90% match rate
```text theme={null}
Find all Series A fintech startups in Europe that raised
between $5M and $20M in the last 18 months
```
## What success looks like
| Metric | Value | Impact |
| :--------------- | :---------------- | :--------------------------------------- |
| **Time Saved** | 20-30 min/record | Weeks of research completed in minutes |
| **Data Sources** | 1,000+ per result | Comprehensive validation |
| **Match Rates** | 90-100% | Industry-leading accuracy |
| **ROI** | 10-20x | Each lead worth \$10-20 in research time |
## What to Avoid
### Overly Broad Queries
**Avoid:** "Find all companies"\
**Better:** "Find all software companies in Boston with 50 to 200 employees"
### Too Many Constraints
**Avoid:** Queries with more than 5-6 different criteria\
**Better:** Focus on 2-4 most important criteria
### Avoid vague terms
**Avoid:** "Find all big companies"\
**Better:** "Find all companies with revenue greater than \$100 million"
### Individual Person Searches Without Context
**Avoid:** "Find all people named John"\
**Better:** "Find all executives named John at Fortune 500 companies"
## FAQ
Deep Lookup provides structured, table-ready data from 1,000+ sources. Unlike search engines that give you links or AI that generates text, we deliver verified, actionable business data you can immediately use.
Skipped results indicate entities that don't match all your filter criteria. To get more results, try removing some filters or making your criteria less restrictive. Remember, you only pay for matched results, not skipped ones.
Yes! In Preview Mode, you can adjust column types, add new columns, and modify specifications before running your full research.
95%+ accuracy with full source transparency. Every data point is validated across multiple sources before delivery.
Yes! Preview Mode gives you 10 free sample records for any query to validate quality before running a full research project.
## Ready to Start Your Research?
Start with 5 free queries to experience the power of Deep Lookup. No credit card required.
[Get Started Free](https://brightdata.com/products/deep-lookup) | [View Pricing](/datasets/deep-lookup/pricing) | [API Documentation](/api-reference/deep-lookup)
# Deep Lookup - pricing
Source: https://docs.brightdata.com/datasets/deep-lookup/pricing
Bright Data Deep Lookup pricing is pay-per-result: you pay only for successfully matched records. Skipped or unmatched results are free. Achieves 95% accuracy.
## Pricing Structure
**\$1.00** per matched record
* 10 enrichment columns included
* Additional columns at \$0.05 each
* 95%+ accuracy
* Source transparency
* Post-run enrichment
* Real-time data extraction
* No charge for skipped results
## How Our Pricing Works
Define your research criteria and choose enrichment columns (first 10 included)
View the worst-case cost before running - full transparency, no surprises
You're only charged for successfully matched records, typically 40-60% less than the maximum
## Understanding Skipped Results & Pricing
**Important:** You never pay for skipped results. When Deep Lookup filters out entities that don't match your criteria, those skipped records don't count toward your bill. This ensures you only pay for data that exactly meets your requirements.
## Pricing Calculator
### Example: Finding 500 B2B Leads
**Your Research:**
* Target: 500 SaaS companies
* Enrichments: 5 columns (all included)
**Results:**
* Records searched: 500
* **Actual matches: 300**
* Skipped (not charged): 200
* Maximum estimate: \$500
* **You pay: \$300**
* **You save: \$200 (40%)**
### Example: Deep Competitor Research
**Your Research:**
* Target: 200 competitors
* Enrichments: 15 columns (10 included + 5 extra)
**Results:**
* Records searched: 200
* **Actual matches: 100**
* Skipped (not charged): 100
* Maximum estimate: \$250
* **You pay: \$125**
* **You save: \$125 (50%)**
## How volume discounts work
Scale your research with progressive savings:
| Monthly Volume | Discount | Your Price |
| :------------------- | :------- | :------------ |
| 1-1,000 records | - | \$1.00/record |
| 1,001-5,000 records | 20% off | \$0.80/record |
| 5,001-10,000 records | 30% off | \$0.70/record |
| 10,000+ records | Custom | Contact sales |
## How Deep Lookup compares for ROI
| Alternative | Cost | Time | Quality |
| :-------------- | :---------- | :------------ | :---------------- |
| Manual Research | \$30-50/row | 20-30 min/row | Variable |
| **Deep Lookup** | **\$1/row** | **Instant** | **95%+ accuracy** |
### Cost Breakdown Example
**Traditional Manual Research:**
* Analyst hourly rate: \$60-100/hour
* Time per record: 20-30 minutes
* Cost per record: \$30-50
* 100 records = \$3,000-5,000
**Deep Lookup:**
* Cost per record: \$1
* Time per record: Instant
* 100 records = \$100
* **Savings: \$2,900-4,900 (97% cost reduction)**
## What every project includes
### Every Research Project Includes
* Real-time data extraction
* 1,000+ sources per result
* Natural language queries
* CSV/JSON export
* Preview mode (10 free samples)
* Source transparency
* Post-run enrichment
### Which enrichments are popular
**Included (first 10 columns):**
* Company name & website
* Industry & description
* Location & headquarters
* Employee count
* Revenue estimates
* Contact emails
* Phone numbers
* Social profiles
* Technology stack
* Funding information
**Additional enrichments (\$0.05 each):**
* Executive names & titles
* Competitor analysis
* News mentions
* Product details
* And 100+ more options
## Frequently asked billing questions
We show the worst-case scenario for full transparency. You'll only pay for actual matches, typically 40-60% less.
You never pay for skipped results. These are entities that didn't match your filter criteria and are automatically excluded from billing.
Yes! Add new columns anytime at \$0.05 per column per matched record.
Never. You only pay for matched records. No setup fees, minimums, or charges for failed matches or skipped results.
You get 5 free queries to test Deep Lookup. Each query can search up to 100 records.
We offer a satisfaction policy. If results don't meet expectations, contact support for resolution.
## Ready to Start?
* \- 5 queries included
* \- Up to 100 records each
* \- No credit card required
*
* \- No minimum commitment
* \- Cancel anytime
* \- Usage-based billing
*
* \- Volume discounts
* \- Custom pricing
* \- Dedicated support
* \- API access
# Customization and filtering
Source: https://docs.brightdata.com/datasets/marketplace/customization-and-filtering
Customize fields and apply filters to Bright Data Marketplace datasets (250+ domains) using the UI or API to get the data subset you need.
Narrow down Bright Data Marketplace datasets (250+ domains) to the exact records and fields you need using the UI or API by creating filtered subsets and selecting the fields to include.
**What you can do:**
* **Customize fields:** choose which columns (fields) to include in your dataset view or export.
* **Filter a dataset:** create a saved subset using filter rules.
**Permissions:** You need access to the dataset and permission to create subsets.
## How to customize fields
Choose which fields (columns) appear in your Bright Data Marketplace dataset view and exports.
1. Navigate to the dataset you want to work with.
2. Open the Fields selector.
3. Select the fields you want to include in the view.
4. Click Apply, and save the view if prompted.
**Expected result:** The dataset preview and any export from this view include only the selected fields.
## How to filter a dataset
Create a saved subset of a Bright Data Marketplace dataset by applying filter rules in the UI or through the API.
### Filter a dataset using the UI
1. Navigate to the dataset you want to filter.
2. Click the Filter icon (top right).
3. Enter a name so you can find this subset later.
4. Under Include filters, add one or more filters (for example: country, job title, date).
5. Click Create subset.
**Expected result:** A new subset appears with only the records that match your filters.
### Filter operators reference
#### Select
Match one or more exact values from a predefined list (for example: countries or regions).
#### Boolean (true/false)
Filter fields that can only be true or false (for example: a field like `verified`).
#### Date
Filter records within a specific date range (start date and end date).
#### Number (operators)
* **Is:** match an exact numeric value.
* **Not:** exclude a numeric value.
* **Exists:** include only records where the field is not empty.
* **List (exact match):** match any value in a provided list.
* **Lower than / Lower or equal to:** match values below (or up to) a threshold.
* **Greater than / Greater or equal to:** match values above (or at) a threshold.
#### String
Filter text fields using the match types available in the UI (for example: exact match or contains).
#### Array
Use Array includes to match records where a multi-value (array) field contains a specific value (for example: categories, attributes or labels).
#### Upload a CSV list
If you need to match many values, use the CSV upload option in the value input. Upload a CSV with one value per row. After upload, the filter matches records where the array contains any value from the uploaded list.
#### CSV upload limitations
* **CSV format:** upload a `.csv` file with one value per row (single column).
* **Maximum 10,000 values per list:** to match more, split the values across multiple CSV lists, up to 100,000 values total, and apply them as filters in separate subsets.
* **No empty or whitespace-only lines:** lines that contain only spaces are rejected.
* **Single column only:** the CSV must contain only one column. If you include a header, it must be a single column.
* **One value per row:** put exactly one filter value on each row. Do not comma-separate multiple values on the same row.
### Includes vs List (exact match)
#### Includes
Use Includes to match records where the field contains the value you enter (partial match).
Example: if the field is `name` and you filter with Includes = `john`, you match values like `John Smith` and `Johnson`.
#### List (exact match)
Use List (exact match) to match records where the field value is exactly one of the values in your list (no partial matches).
Example: List (exact match) = `John Smith`, `Jane Doe` matches only those exact values.
### Group filters (rule-based filters)
1. Click + Add filter.
2. Select Add group.
3. Define your group rules (for example: category is "Electronics" and brand is "Dell" or "Apple").
**Expected result:** The subset includes only records that match your group logic.
### Limitations
* Groups can't be nested.
* A maximum of two groups per filter is allowed.
* A maximum of four inputs per filter group is allowed.
* To filter by more values, use a CSV list upload option (if available) or contact your account manager.
* For more complex queries, contact your account manager.
## Troubleshooting
* **I can't see the Filter icon.** Make sure you're viewing the dataset table and that your account has permission to create subsets.
* **My subset returns no results.** Remove filters one by one to identify the restrictive condition, then reapply the correct values and operators.
## Next steps
* [Export or download a dataset subset](/datasets/marketplace/data-delivery-and-export)
* [Filter a dataset by API](/datasets/marketplace/filter-dataset-by-api)
* [Bright Data Marketplace datasets overview](/datasets/marketplace/overview)
# Marketplace data delivery
Source: https://docs.brightdata.com/datasets/marketplace/data-delivery-and-export
Receive Bright Data Marketplace datasets in JSON, NDJSON, CSV, XLSX or Parquet via email, API, webhooks or cloud destinations like S3 and Snowflake.
Once you purchase or subscribe to a dataset, Bright Data delivers it directly to your preferred destination. Choose from 9 delivery methods and 5 output formats to fit your existing infrastructure and workflow.
## Output formats
Datasets are available in the following formats:
| Format | Description |
| ------- | ------------------------------------------- |
| JSON | Standard structured format |
| NDJSON | Newline-delimited JSON, ideal for streaming |
| CSV | Spreadsheet-compatible format |
| XLSX | Microsoft Excel format |
| Parquet | Columnar format optimized for analytics |
You can also receive data in compressed (gzip) format to reduce file size.
## Which delivery methods are supported
Choose how and where your data is delivered:
| Method | Description |
| -------------------- | ---------------------------------------------------- |
| Email | Receive datasets directly in your inbox |
| API download | Download via the Bright Data API using a snapshot ID |
| Webhook | Get data pushed to your endpoint automatically |
| Amazon S3 | Deliver directly to your S3 bucket |
| Google Cloud Storage | Deliver to your GCS bucket |
| Google Cloud Pub/Sub | Stream data via GCP Pub/Sub |
| Microsoft Azure | Deliver to your Azure Blob Storage container |
| Snowflake | Load directly into your Snowflake data warehouse |
| SFTP | Deliver via secure file transfer protocol |
## How to set up delivery
After purchasing a dataset, go to **Control Panel → My Datasets**.
Select your dataset and click **Delivery Settings**.
Pick your preferred delivery method and output format.
Enter the destination credentials (bucket name, webhook URL, SFTP host, and so on).
Click **Save**. Your data is delivered automatically on each refresh.
## Related
* [Purchase options](/datasets/marketplace/purchase-options)
* [Pricing](/datasets/marketplace/pricing)
* [Filter dataset by API](/datasets/marketplace/filter-dataset-by-api)
# Dataset view
Source: https://docs.brightdata.com/datasets/marketplace/dataset-view
Browse dataset details in the Bright Data Marketplace (250+ domains), including data fields, freshness, sample downloads and purchase options.
## How to preview a data sample
* Data-fields
* Records
* Freshness
* Min order
* Download data sample: JSON, CSV, number records (display 30 records but you get 1,000 records)
* CSV/JSON
* Create a subset
* Contact us
* Purchase options
### How to customize a dataset
* Customize - Create a custom subset by hiding/showing and renaming fields and filter the dataset according to your requirements.
* Customize Fields
* Filter
## Where to view the data dictionary
* Search attribute
* Table
* Column name
* Description
* Data type
## When to use this dataset
* Case study
* Related subset
* What category it’s related to
## What this dataset covers
* Description about the dataset
* What domain we collect from
* Use case categories
* Customize dataset CTA
* Delivery options
* FAQ
## Initiate by API
* Ability to submit a request through the API or the UI to collect fresh data based on your own URLs (inputs)
* +Add input
* Upload CSV
* Creating API key to initiate a collection
* Setting the expected records
* Data Collection Choices
* Discover new records
* Specific URL collection
* Purchase options
# Dataset marketplace FAQs
Source: https://docs.brightdata.com/datasets/marketplace/faqs
FAQs about Bright Data's Dataset Marketplace (250+ domains), including available datasets, delivery options and purchase choices for ready-to-use data.
The following is a partial list of datasets available for immediate download from the Datasets Marketplace:
**Popular Datasets**
* Amazon products
* Crunchbase companies information
* Facebook - Posts by group URL
* Github repository
* Glassdoor companies overview information
* LinkedIn company information
* LinkedIn people profiles
* LinkedIn posts
* Zillow properties listing information
The datasets are broken up by category. Here are the main categories and some of the datasets:
**eCommerce Data**
* amazon.com, amazon.co.uk, amazon.de, amazon.es, amazon.fr, amazon.in, amazon.it,
* homedepot.com, homedepot.ca
* lazada.com.my, lazada.sg, lazada.vn
**Real Estate Data**
These datasets include information regarding housing data, real estate prices, rent prices etc.
* Bayut UAE Property Listings
* Booking.com Property Listings
* Dubizzle UAE Property Listings
* PropertyFinder Property Listings
* US Consumer Property
* ZoomProperty UAE Property Listings
* infocasas.com.uy
* inmuebles24.com
* metrocuadrado.com
* otodom.pl
* properati.com.co
* realestate.com.au
* toctoc.com
* zillow\.com
* zonaprop.com.ar
* zoopla.co.uk
**Social Media Data**
* facebook.com
* instagram.com
* linkedin.com
* pinterest.com
* reddit.com
* tiktok.com
* unashamedcataddicts.quora.com
* vimeo.com
* x.com
* youtube.com
**Travel Data**
* Booking.com Hotel Room Pricing and Availability
* Deliveroo Restaurant Listings
* OpenTable Restaurant Listings
* Short-Term Rental Occupancy & Pricing Dataset
* Talabat Restaurant Listings
* Tripadvisor Restaurant Listings
* Zomato UAE Restaurant Listings
* airbnb.com
**B2B Data**
* Business Contacts Dataset
* Business Firmographic Data
* Business Intelligence Dataset
* Business Location (POI) Dataset
* Companies Hierarchy Dataset
* Online Intent Data
* Politically Exposed Persons List
* Tech Install base Data Feed
* US B2B Employees
* US Consumer Demographics
* crunchbase.com
* g2.com
* glassdoor.com
* google.com
* indeed.com
* linkedin.com
* manta.com
* owler.com
* slintel.com
* stackoverflow\.com
* trustpilot.com
* ventureradar.com
* xing.com
* yelp.com
The datasets marketplace is continously updated with fresh datasets. For the complete list, click on "Web Data" on the sidebar, and then on "Datasets Marketplace" on the top bar.
If the domains you need aren't exist in the Marketplace, you can request them through the Custom Dataset (CDS).
Yes! you can download a few free datasets:
* espn.com - NBA data
* goodreads.com
* imdb.com
* worldpopulationreview\.com
The datasets marketplace is continously updated with fresh datasets. For the complete list, click on “Web Data” on the sidebar, and then on “Datasets Marketplace” on the top bar.
The schedule run is designed to ensure timely delivery.
The delivery deadline is calculated based on previous collection cycles and the estimated refresh duration.
Therefore, the collection may start earlier than the delivery date to guarantee that the data is delivered on time.
You can find your data snapshots under the "My datasets" tab. There, you'll see a table with information about each snapshot, including its status: ready, failed, or in building.
A Snapshot ID is a unique identifier assigned to a specific data snapshot, formatted as "snap\_XXXXXX".
You should use the Snapshot ID whenever there is an issue with a particular data snapshot. Including this ID in your support ticket helps the support team quickly identify the exact snapshot in question, leading to faster issue resolution.
The Snapshot ID ensures that both you and the support team are referring to the same data set, reducing confusion and delays in addressing your problem.
You can set a record limit in two ways:
**Using the control panel:** Before purchasing a dataset, click “Proceed to purchase.” On the “Choose delivery frequency” page, select the “Too pricey? Limit dataset records” option to specify your desired record limit.
**Through the Filter API:** Add a parameter to limit the number of records returned by the API. For reference, see: [Dataset Filter API - records\_limit](/api-reference/marketplace-dataset-api/filter-dataset#body-records-limit).
Currently, there is no monthly commitment or minimum order of \$250 required when using the Dataset Filter API. You only pay based on your actual record consumption.
When you submit a dataset filter API request, compute resources are used to identify records matching your filter criteria. If matching records are found, you will be charged based on the amount of these matched records. However, if no matching records are found, you will not be charged. To avoid charges while exploring your filter criteria, you can test filters through the dataset preview table in the control panel, which offers up to 10 free filters per day.
Some fields may have lower fill rates due to limitations or gaps in the publicly available source data. Fill rates vary depending on dataset type and source quality - which can result in partial coverage for specific attributes. We provide detailed fill rates and statistics for each dataset to help you evaluate completeness before purchasing.
Bright Data offers several services for accessing and managing datasets:
1. **Dataset Marketplace**: This is a centralized platform where you can discover, customize, and purchase high-quality datasets from over 250 domains. You can browse pre-built datasets across multiple domains, examine data samples, and apply advanced filters. [Explore the Dataset Marketplace here](https://brightdata.com/datasets/marketplace/browse).
2. **Dataset APIs**: These APIs allow you to request, initiate, and manage data collections. You can define parameters for new dataset collections, check the status of your requests, and download datasets using snapshot IDs. [Learn more about Dataset APIs here](https://brightdata.com/api-reference/marketplace-dataset-api/request-a-collection).
3. **Deep Lookup**: This service provides a more granular and streamlined way to request and manage data collections, facilitating effective dataset generation according to your specific needs. [Explore Deep Lookup here](/datasets/deep-lookup/overview).
Would you like more information on how to use any of these services?
* By default, standard LinkedIn profile records **do not** include email addresses or phone numbers. This information is not publicly available on LinkedIn.
* However, Bright Data offers an **enriched business contact** solution (in partnership with RevenueBase) that adds business emails and phone numbers for many LinkedIn people profiles, fully GDPR-compliant and sourced via third-party validation.
* Contact data coverage may vary by profile and use case.
* In the Dataset Marketplace, after selecting "LinkedIn People Profiles", use the **Contact filters** button (on the right side of the Data sample view) to choose your contact data options:
* **Standard LinkedIn profile data:** No contact info.
* **Enriched business contact info:** Select “Standard Profiles + Enriched with Business Contact Info” or “Only Profiles with Business Contact Info” to receive available business emails and phone numbers (where provided via RevenueBase partnership and in accordance with GDPR/compliance).
* Click “Apply filter” to preview and purchase the dataset with your chosen contact enrichment.
Yes. All provided business contact data is sourced and processed according to GDPR and other compliance requirements, using approved partners such as RevenueBase.
Yes. Bright Data’s [Deep Lookup](https://brightdata.com/cp/deep-lookup) can search for people and return available business contact details (email/phone), where legally sourced and compliant. Specify your entity and required columns in the query (e.g., email, phone).
* Go to Control Panel → Dataset Marketplace → LinkedIn People Profiles.
* Click “Preview sample” to review all available fields.
* For enriched datasets, use the **Contact filters** panel as described above, and preview sample rows before placing your order.
For full details on compliance, permissible usage, and supported geographies, speak directly with your Bright Data account manager or reach out via [Support](https://brightdata.com/cp/support).
**Summary:**
* **Standard LinkedIn profiles** don’t include emails/phones.
* **Enriched business contact info** (email/phone) is available: just use the Contact filters button in the Dataset Marketplace view.
* **Deep Lookup** is another route for contact discovery.
* **Always review** the filtered sample before purchasing, and contact support for custom requirements.
Let me know if you want a live demo, pricing, or coverage estimate!
The Filter API rate limit is 120 requests per hour. This applies to all API calls and snapshot triggers within the specified time frame.
Note: Plan your API calls accordingly to stay within the hourly limit. Consider implementing retry logic with exponential backoff for optimal performance.
You can send up to 10,000 input lines in a single API request when using list filters or include filters.
Best Practice: For large datasets, consider batching your requests to stay within the 10,000 line limit while maintaining efficient processing.
The maximum input file size is 200 MiB for any single API request.
Warning: Files exceeding 200 MiB will be rejected. Compress your data or split large files into smaller chunks before submission.
You can download snapshots up to 5 GB as a single file.
For snapshots larger than 5 GB, the API will automatically provide chunked download options or streaming capabilities to handle the data efficiently.
### Quick Reference
| Limit Type | Value | Description |
| :---------------- | :------- | :------------------------------------- |
| Rate Limit | 120/hour | Maximum API calls per hour |
| Input Lines | 10,000 | Maximum values in list/include filters |
| Input File Size | 200 MiB | Maximum size for uploaded files |
| Snapshot Download | 5 GB | Maximum size for single-file download |
**Need Higher Limits?**\
Contact our Enterprise team for custom rate limits and increased capacity options tailored to your business needs.
Snapshot IDs are returned when you [Trigger Collection](/api-reference/rest-api/scraper/asynchronous-requests) (POST `/datasets/v3/trigger`), [Filter Dataset](/api-reference/marketplace-dataset-api/filter-dataset-with-csv-json-files) (POST `/datasets/filter`), or via a dataset subscription. You can also [list all your snapshots](/api-reference/marketplace-dataset-api/get-dataset-list) with GET `/datasets/v3/snapshots`.
> See "[Before you begin](/api-reference/marketplace-dataset-api/deliver-snapshot#before-you-begin)" for more details.
The `id` in the response is a delivery job ID. Use it to track delivery progress by calling GET `/datasets/v3/delivery/{delivery_id}`. Poll until status is "done".
> See [Tracking delivery status](/api-reference/marketplace-dataset-api/deliver-snapshot#tracking-delivery-status) for more details.
Yes. The snapshot must be in `ready` status. Check with GET `/datasets/snapshots/{id}` before calling deliver.
> Possible statuses: `scheduled`, `building`, `ready`, `failed`.
Yes. Call this endpoint multiple times with different delivery configurations for the same snapshot ID.
json, jsonl, and csv.
Use the `batch_size` parameter to set the number of records per file. Each file (batch) must stay under the 5GB hard limit. Estimate `batch_size` by dividing 5GB by your average record size, then start lower and adjust based on the actual file size you get.
For example, if your average record is \~5KB, a `batch_size` of 1,000,000 records lands right at the \~5GB limit. Start at 500,000 records (\~2.5GB) to stay safely under, then raise or lower `batch_size` based on the file size you receive.
The most common cause is that your `batch_size` produces a file larger than 5GB. For example, if your average record size is \~5KB, a `batch_size` of 1,000,000 produces a \~5GB file that may exceed the limit. Lower your `batch_size` (e.g., to 100,000) and retry.
Yes. Set `compress: true` to receive gzip-compressed files.
5GB. This is a hard limit per delivered file. Use `batch_size` to control how many records go into each file and ensure each stays under this threshold.
# Marketplace fill rates
Source: https://docs.brightdata.com/datasets/marketplace/fill-rates-and-statistics
Review per-field fill rates, record counts and data freshness on the Statistics tab of any Bright Data Marketplace dataset (250+ domains) before purchase.
Every Bright Data Marketplace dataset page exposes a Statistics tab with field-level fill rates, the total record count and a freshness indicator. Review these before purchase to confirm the dataset meets your requirements.
## What are fill rates?
A fill rate is the percentage of records in a dataset where a specific field is populated. A fill rate of 100 percent means every record contains a value for that field. A lower fill rate means some records are missing values for that field.
Fill rates vary depending on:
* The dataset type (social media, B2B, eCommerce)
* The source quality and what is publicly available
* The specific field being measured (email, phone number, location)
## Where to find fill rates
Fill rates and statistics are shown on each dataset's page in the Marketplace.
In the Bright Data Control Panel, go to **Dataset Marketplace**.
Open the dataset whose coverage you want to review.
Navigate to the **Statistics** tab to view field-level fill rates and record counts.
## What statistics are available?
For each dataset you can review:
* **Fill rate per field**: percentage of records with a value
* **Record count**: total number of available records
* **Freshness**: how recently the data was collected
## Why do some fields have low fill rates?
Some fields have lower fill rates because of limitations or gaps in the publicly available source data. For example, phone numbers and email addresses are not always publicly listed, so contact fields commonly show lower fill rates than name or location fields.
Always review fill rates before purchasing to confirm the data meets your specific requirements.
# Filter dataset by API
Source: https://docs.brightdata.com/datasets/marketplace/filter-dataset-by-api
Filter Bright Data Marketplace datasets programmatically using the Filter Dataset API. See the full API reference for parameters and examples.
See [Filter Dataset](/api-reference/marketplace-dataset-api/filter-dataset) API Reference page
# Dataset marketplace overview
Source: https://docs.brightdata.com/datasets/marketplace/overview
Browse and buy 350+ ready-to-use datasets from 250+ domains. Updated on a schedule, customizable by field and delivered in your preferred format.
## Introduction to the Dataset Marketplace
The Dataset Marketplace is a one-stop platform for discovering, customizing and purchasing high-quality datasets from over 250 domains. Datasets are regularly updated and ethically sourced from publicly available information, ensuring both accuracy and compliance. Whether you need ready-to-use or on-demand solutions covering social media, real estate, B2B data or AI training, flexible purchase options and field-level customization give you the exact data you need.
## Get started
Browse a wide range of pre-built, validated datasets across multiple domains.
Examine data samples, data fields and customization options before purchase.
Refine datasets with advanced filters, custom field selection and rule-based subsets.
Preview and download a sample of any dataset in multiple formats.
Review data completeness and field coverage to ensure accuracy.
Choose between one-time purchases, subscriptions and flexible data delivery models.
Understand per-record costs, compute fees and available pricing plans.
Learn how to receive datasets via email, API, webhooks and cloud services including S3, Azure, Google Cloud, Snowflake and SFTP.
# Pricing
Source: https://docs.brightdata.com/datasets/marketplace/pricing
Understand how Bright Data Marketplace datasets are priced, including per-record costs and compute fees for pre-collected and fresh records.
* Pre-collected records
* Fresh records
* Pricing is determined based on compute cost and record cost to reach and collect the data.
For delivery methods and output formats, see [Data delivery and export](/datasets/marketplace/data-delivery-and-export).
# Purchase options
Source: https://docs.brightdata.com/datasets/marketplace/purchase-options
Compare 2 Bright Data Marketplace purchase options: one-time or subscription, pre-collected or fresh data, with supported delivery destinations.
* Purchase options → Flexibility
* One-time purchase
* Subscription
* Select data type
* Pre-collected data - Access data that was collected recently and is ready for use.
* Fresh data - Access data that's up-to-date and fresh, available immediately after it's collected.
* Data distribution - 544M records available spanning from Apr 12 2021, to Feb 27 2024
* Specific time range - Access data from a specific time range.(back from today)
* Available only for pre-collected.
* Select Period
* Collect fresh data for the rest of the records
* Records outside of the specified time range will be scraped when the request is processed and will incur an additional fee. The refresh time depends on the size of the dataset. You'll get an email when your snapshot is ready.
* As the data changes, these records may no longer match your filters and may be excluded from your dataset. If these records later do not meet your filters, they will be removed, but you will still be charged for the initial scrape.
* Set record limit - Enter the number of records you would like. Records are selected randomly. Notice that the price will change according to the record count.
* Order
* Dataset name - “LinkedIn people profile dataset”
* Total amount of records - 544M records
* “Records” - cost of total records
* Number of records - 544M records
* Cost per record - \$0.0006
* Subtotal - (before VAT or salestax)
* Total
* Price adjustment (max snapshot cost)
* Small dataset fee (min snapshot cost)
# Build a scraper with the AI Agent
Source: https://docs.brightdata.com/datasets/scraper-studio/ai-agent
Use Bright Data Scraper Studio's AI Agent to build a custom web scraper from a natural-language prompt in about 10 minutes, no coding required.
Bright Data Scraper Studio's AI Agent builds a custom web scraper from a natural-language description. Give it a target URL and describe the data you want; the agent generates an output schema, writes the scraper code, and hands you a runnable scraper you can trigger by API, manually, or on a schedule. This tutorial walks through the full flow.
**Time to complete:** about 10 minutes
Prefer the terminal? Use the [Bright Data CLI](/datasets/scraper-studio/build-with-the-cli) to build the same scraper from any shell or coding-agent embedded terminal (Claude Code, Cursor, Codex).
## Prerequisites
* A Bright Data account ([sign up free](https://brightdata.com/?hs_signup=1\&utm_source=docs))
* The URL of the website you want to scrape
## Build your first scraper with the AI Agent
In the Bright Data control panel, click **Scrapers** in the left menu and open **Scraper Studio**.
Paste the URL of the page you want to scrape into the chat input. Along with the URL, add any context that helps the AI build a more accurate scraper on the first try. The more context you provide, the better the generated code.
Useful context to include:
* **Specific fields** you need: "I need price, title, and stock status"
* **Where the data lives** on the page: "prices are in the product detail panel, not the listing page"
* **Actions required** to reach the data: "click 'Show more' to load full descriptions"
* **CSS selectors**, if you know them: `.product-price span.amount`
* **Page load behavior**, if the site is slow or lazy-loads content: "results load dynamically, give it extra time"
> **Expected result:** the AI Agent acknowledges the URL and may ask one or two clarifying questions about the data you want.
Respond in plain language.
> **Expected result:** the AI Agent generates a schema, a structured list of fields with data types that will become your scraper's output.
Read through the generated schema. You have four options:
* **Approve:** click Approve to accept the schema as-is
* **Decline:** type feedback in the chat (for example, "Remove the image field and add a rating field") and the AI regenerates the schema
* **Edit inline:** modify the schema directly without going back to the chat
* **Upload your own schema:** bring your own schema file; download the example file to see the correct format
Inline editing options:
* **Edit a field** (pencil icon): change a field's name or data type
* **Delete a field** (trash icon): remove fields you do not need
* **Add a field** (plus button): add new fields to the schema
Additional controls:
* **Start from scratch:** clears every field so you can build the schema manually from an empty state
* **Reset the schema:** discards inline changes and returns to the original AI-generated schema
Click **Approve** when you are ready.
> **Expected result:** once approved, the AI Agent starts generating the scraper code.
The AI writes the full scraper, including extraction logic, navigation handling, data validation, and error handling. This takes a few minutes.
> **Expected result:** a confirmation popup appears indicating your scraper is ready.
Click **Try it out** to open the Initiate Manually page. Review the collection settings and click **Start** to begin data collection.
You can also choose an alternative initiation method:
* **[Initiate by API](/api-reference/scraper-studio-api/ai-flow/overview):** trigger the scraper programmatically without opening the control panel
* **Schedule:** run the scraper on a daily, weekly, or custom interval
> **Expected result:** the scraper collects data. Monitor progress from the Runs dashboard and download results in JSON, NDJSON, CSV, or XLSX once the job completes.
After the scraper is ready, use the [Self-Healing tool](/datasets/scraper-studio/self-healing-tool) to make plain-language changes, or open the code in the Bright Data Scraper Studio IDE to edit it directly.
## What can the AI Agent build?
Reach for the Bright Data Scraper Studio AI Agent when the data you need isn't already in the [Datasets Marketplace](/datasets/marketplace). For sites Bright Data already covers (Amazon, Walmart, LinkedIn, eBay), the marketplace ships pre-built scrapers that are faster and cheaper than building your own. The AI Agent shines on regional ecommerce, B2B catalogs, niche verticals and any site where no marketplace scraper exists.
The AI Agent builds one of five scraper types. Pick the one that matches what you have and what you need.
| Scraper type | What you provide | What you get | Page visits per input |
| ------------------- | ---------------------------- | ------------------------------------------------- | --------------------- |
| **PDP** | List of product URLs | Full per-product detail | 1 |
| **Discovery** | Category / listing URL | Listing-level rows (title, price, rank) | 1 |
| **Discovery + PDP** | Category / listing URL | Full per-product detail for every item | 1 + N |
| **Search** | Keyword (+ optional country) | Either Discovery or Discovery + PDP shape | 1 + M |
| **Sitemap** | Domain or `sitemap.xml` URL | Full per-page detail for every URL in the sitemap | 1 + N |
**Not a crawler.** Don't pass a homepage and ask for "everything." Each scraper is scoped to one data shape.
### When do I use a PDP scraper?
Use a PDP scraper when you have a list of specific product URLs and need full per-product detail. Each input URL produces one row.
**Example:** 100 product URLs from a regional retailer like [dm.de](https://www.dm.de) → 100 rows with title, price, availability, images and any other field you defined in the schema.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a PDP scraper for dm.de. For each product URL I provide, extract the product title, price, availability, brand, rating and all product image URLs. Return one row per input URL.
```
### When do I use a Discovery scraper?
Use a Discovery scraper when you need an overview of items on a single listing page and can live without per-product detail. Each input URL produces N rows where N equals the number of items in the listing.
**Example:** A category page like `https://www.dm.de/baby-und-kind` → 30 rows with title, price, rating and listing position. No description, no full image gallery.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Discovery scraper for the dm.de category page https://www.dm.de/baby-und-kind. Return one row per item shown in the listing with title, price, rating and listing position. Do not open the individual product pages.
```
### When do I use a Discovery + PDP scraper?
Use a Discovery + PDP scraper when you need full per-product detail for an entire category. Each input URL produces N rows with the full PDP shape.
This is the most expensive type. One category URL with 200 products costs roughly 200 times a single PDP scrape.
**Example:** A category page on a regional retailer like [decathlon.fr](https://www.decathlon.fr) → full per-product detail for every item in that category.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Discovery + PDP scraper for a category page on decathlon.fr (for example a sport category like running shoes). Find every product in the category, open each product page, and extract full detail: title, price, availability, description, rating and image URLs. Return one row per product.
```
### When do I use a Search scraper?
Use a Search scraper when you do not have specific URLs. Provide a keyword and optionally a country; the AI Agent picks Discovery or Discovery + PDP shape based on whether you asked for listing-level fields or full PDP detail.
**Example:** keyword "brake pads" on a B2B parts site like [autodoc.de](https://www.autodoc.de) → matching product results in either listing-level or full PDP shape.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Search scraper for autodoc.de. For the keyword "brake pads" in Germany, return matching products with title, price, brand and product URL. No need to open each product page.
```
### When do I use a Sitemap scraper?
Use a Sitemap scraper when you need data from many pages on a site and the site publishes those page URLs in an XML sitemap. The sitemap lets the scraper discover URLs directly instead of clicking through category pages, pagination or search results, so it fits large ecommerce sites, marketplaces, catalogs, blogs and documentation sites.
Provide a domain or a sitemap URL. Each sitemap produces N discovered page URLs, and the scraper visits each relevant page to collect the fields you defined in the schema.
**Example:** a domain like [dm.de](https://www.dm.de) or its sitemap at `https://www.dm.de/sitemap.xml` → discover every product URL in the sitemap → visit each product page → collect full product detail.
Paste this into the AI Agent chat:
```text Prompt theme={null}
Build a Sitemap scraper for dm.de. Collect all product URLs from the sitemap, visit each product page, and extract product name, price, SKU, description, image URL and availability. Return one row per product.
```
If you already have the sitemap URL, point the AI Agent straight at it:
```text Prompt theme={null}
Build a scraper using this sitemap: https://www.dm.de/sitemap.xml. Extract all product page URLs, visit each product page, and collect the product detail fields.
```
## Frequently asked questions
Yes. Every scraper the AI Agent generates can be opened in the Bright Data Scraper Studio IDE and edited directly. If you prefer not to write code, use the [Self-Healing tool](/datasets/scraper-studio/self-healing-tool) to request changes in plain language.
The AI Agent generates scrapers that run on Bright Data's proxy and unblocking infrastructure, which handles most anti-bot defenses. For sites that require a logged-in session, build the scraper in the IDE and use `set_session_cookie()` or the authentication pattern that matches the target site.
The AI Agent relies on the context you give it. If the output is off, decline the schema and add more specifics about field names, selectors, or the exact page section where the data lives. You can also use the [Self-Healing tool](/datasets/scraper-studio/self-healing-tool) to refine a generated scraper after the fact.
## Related
Build the same scraper from the terminal or any coding agent
Build a scraper by writing JavaScript directly
Update a generated scraper with plain-language prompts
# Web scraping basics in Scraper Studio
Source: https://docs.brightdata.com/datasets/scraper-studio/basics-of-web-scraping
Understand how Scraper Studio IDE scrapers are structured, from inputs and interaction code to parsing, stages, workers, and output records.
Bright Data Scraper Studio IDE scrapers are built from a few core parts: **inputs**, **interaction code**, **parser code**, **stages**, **workers**, and **output records**.
This page explains how those parts work together so you can understand, edit, and debug scraper code with confidence.
For a step-by-step build walkthrough, see [**Develop a scraper with the IDE**](/datasets/scraper-studio/develop-a-scraper).
## Prerequisites
* Basic JavaScript familiarity (variables, functions, async control flow)
* An active [Bright Data account](https://brightdata.com/)
## **Basic scraper flow**
A Scraper Studio IDE scraper usually follows this flow:
1. The scraper receives an **input**, such as a URL, keyword, location, or custom value.
2. **Interaction code** uses that input to open a page, send a request, click, scroll, paginate, or create the next stage.
3. **Parser code** extracts structured data from the loaded page or response.
4. The scraper saves records to the output dataset with `collect()` or `set_lines()`.
5. The final output is delivered according to the scraper’s delivery preferences.
## **Inputs**
Inputs are the values passed into a scraper run.
A scraper might use:
* `url`, for page-based scraping
* `keyword`, for search or discovery flows
* `location`, for location-based searches
* `country`, `date`, `category`, or any other custom field
The available input fields are defined in the scraper’s [**input schema**](/datasets/scraper-studio/input-and-output-schema#what-is-the-input-schema).
Interaction code reads these values from the `input` object:
```js theme={null}
navigate(input.url);
```
For a keyword-based scraper:
```js theme={null}
navigate(`https://example.com/search?q=${input.keyword}`);
```
A scraper does not always need external input. For example, a scraper can use a hardcoded URL if it always collects from the same page.
## **Interaction code**
Interaction code controls how the scraper reaches the data.
It can:
* Navigate to a page
* Send HTTP requests
* Wait for page elements
* Click buttons
* Type into forms
* Scroll through a page
* Handle pagination
* Create additional crawl stages
A simple interaction flow looks like this:
```js theme={null}
navigate(input.url);
wait('.product-title');
const data = parse();
collect(data);
```
In this example:
* `navigate(input.url)` opens the page from the input.
* `wait('.product-title')` waits until the expected page element appears.
* `parse()` runs the parser code.
* `collect(data)` saves the extracted record.
Interaction code is responsible for **getting to the right page or response**. It should not contain most of the extraction logic. That belongs in parser code.
## **Parser code**
Parser code extracts structured fields from the page HTML or response.
Parser code commonly uses Cheerio, a jQuery-like API, to read page elements:
```js theme={null}
return {
title: $('h1').text_sane(),
price: $('.price').text_sane(),
availability: $('.stock-status').text_sane(),
};
```
The parser returns a JavaScript object. That object becomes the structured data your scraper can collect.
For example, a parser can return product data, profile data, listings, article content, or links that the interaction code will use in the next stage.
## `parse() `**and** `collect()`
`parse()` and `collect()` connect interaction code, parser code, and final output.
### `parse()`
`parse()` runs the parser code for the current page or response and returns its result.
Example:
```js theme={null}
let data = parse();
```
### `collect()`
`collect()` appends one record to the output dataset.
Example:
```js theme={null}
collect({
title: data.title,
price: data.price,
url: location.href,
});
```
## **Multi-stage scrapers**
Some scrapers need more than one step to reach the final data.
For example:
1. Start from a search page.
2. Discover result pages.
3. Open each result page.
4. Extract final details.
Scraper Studio supports this pattern with **stages**.
A stage is a separate crawl step. `next_stage()` sends new input values to the next stage.
Example:
```js theme={null}
navigate(`https://example.com/search?q=${input.keyword}`);
const results = parse().results;
for (const result of results) {
next_stage({ url: result.url });
}
```
Then the next stage can use the new `url` input:
```js theme={null}
navigate(input.url);
const data = parse();
collect(data);
```
Multi-stage scrapers are useful for:
* Search results → detail pages
* Category pages → product pages
* Listing pages → profile pages
* Pagination flows
* Discovery workflows where one page creates many child pages
Scraper Studio can run stages across workers, so this pattern is usually more scalable than processing every page serially in one long script.
## **Parent and child crawls**
When a scraper uses `next_stage()`, it creates a parent-child relationship between crawls.
For example:
* The search page is the **parent** crawl.
* Each result page created from it is a **child** crawl.
This relationship is useful when debugging. In the crawl inspector, you can see which input or page created another page, inspect child pages, and trace where failures happened in a multi-stage flow.
## **Code workers and Browser workers**
Scraper Studio supports two worker types.
### **Code worker**
A **Code worker** uses HTTP requests and raw responses.
Use a Code worker when:
* The data is available in the raw HTML
* The data is available from a public JSON endpoint
* The page does not require browser interaction
* You want faster and more cost-efficient scraping
Code workers cannot click, scroll, type, or run browser-only functions.
### **Browser worker**
A **Browser worker** uses a real headless browser.
Use a Browser worker when:
* The page renders data with JavaScript
* You need to click, scroll, type, or interact with the page
* You need to wait for elements to appear
* You need to capture browser network traffic
* The site requires browser-like behavior
Start with a Code worker when possible. Switch to a Browser worker when the target data is not available in the raw response or when browser interaction is required.
For the complete comparison, see [**Worker types**](/datasets/scraper-studio/worker-types).\\
## How does Scraper Studio handle blocking and CAPTCHAs?
Scraping at scale can trigger site defenses such as:
* IP blocking
* Rate limits
* CAPTCHAs
* Fingerprinting
* Bot detection
Scraper Studio runs on Bright Data’s proxy and unblocking infrastructure, so you do not need to manage proxy rotation, sessions, or retry logic yourself.
Depending on the scraper configuration and worker type, Scraper Studio can:
* Route requests through Bright Data proxy infrastructure
* Retry blocked requests
* Use browser-like fingerprints for Browser workers
* Support CAPTCHA-solving workflows with `solve_captcha()` when applicable
Your scraper code should focus on reaching the right pages and extracting the required data. Bright Data handles the access infrastructure behind the scenes.
## **How schema fits into scraper structure**
The scraper code and schema work together.
### **Input schema**
The input schema defines what values the scraper can receive.
Example input fields:
```js theme={null}
{
"url": "https://example.com/product/1",
"country": "US"
}
```
Interaction code reads those values from `input`.
### **Output schema**
The output schema defines what fields the scraper returns.
It is usually generated from the records passed to `collect()`.
Example collected record:
```js theme={null}
collect({
title: data.title,
price: data.price,
availability: data.availability,
});
```
Those fields become part of the output schema.
For more details, see [**Input and output schema**](/datasets/scraper-studio/input-and-output-schema).
## **Common scraper patterns**
### **Single-page scraper**
Use this pattern when each input maps to one final page.
Example:
```js theme={null}
navigate(input.url);
collect(parse());
```
Best for:
* Product pages
* Profile pages
* Article pages
* Detail pages
***
### **Search or discovery scraper**
Use this pattern when an input creates multiple result pages.
Example:
```js theme={null}
navigate(`https://example.com/search?q=${input.keyword}`);
const results = parse().results;
for (const result of results) {
next_stage({ url: result.url });
}
```
Best for:
* Search pages
* Category pages
* Directory listings
* Marketplace discovery
***
### **Multi-page detail scraper**
Use this pattern when a scraper must move from a list page to many detail pages.
Example flow:
**Input keyword → Search page → Result URLs → Detail pages → Output records**
Best for:
* Product discovery
* Lead generation
* Local business listings
* Review collection
* Job listings
***
## **Debugging mental model**
When a scraper does not return the expected data, check the flow in order:
1. **Input**, Did the scraper receive the expected input values?
2. **Interaction code**, Did it reach the correct page or response?
3. **Parser code**, Did the selectors extract the expected fields?
4. **Stages**, Did `next_stage()` create the expected child pages?
5. **Output records**, Did `collect()` emit the expected structure?
6. **Schema**, Does the output schema include the returned fields?
7. **Worker type**, Does the page require Browser worker behavior?
This flow helps isolate whether the issue is caused by input data, navigation, parsing, staging, schema, or worker configuration.
## Related
Choose between Browser worker and Code worker
Full reference for interaction and parser commands
Step-by-step walkthrough of building a scraper in the IDE
Recommended patterns for fast, reliable scrapers
# Scraper Studio IDE best practices
Source: https://docs.brightdata.com/datasets/scraper-studio/best-practices
Bright Data Scraper Studio IDE best practices for dead page detection, request batching, pagination, popups, timeouts, retries and parser code in 8 sections.
This guide shows the coding patterns the Bright Data Scraper Studio team recommends for writing fast, reliable scrapers in the IDE. Each section contrasts a common mistake with the preferred pattern and explains the reason.
## How do I detect dead pages reliably?
When using `navigate()`, add a `dead_page()` condition so the scraper does not retry pages that do not exist. Bright Data Scraper Studio automatically marks HTTP 404 responses as dead, but many sites return 200 with a "not found" template, so you must check for that yourself.
Do not wrap `wait()` in a `try/catch` and call `dead_page()` from the `catch` block. A thrown `wait()` only tells you the selector did not appear within the timeout, not that the page is actually dead.
```js Bad theme={null}
try {
// Waits 30 seconds for 'ok-selector' even if the page is already dead
wait('ok-selector');
} catch (e) {
// You cannot prove the page is dead from a wait timeout alone
dead_page("Page doesn't exist");
}
```
```js Good theme={null}
wait('ok-selector, 404-selector');
if (el_exists('404-selector'))
dead_page();
```
## How do I minimize requests to the browser?
Interaction commands like `click`, `type`, `el_exists`, `el_is_visible`, `wait`, and `wait_visible` each send a request to the browser. Combine selectors into a single call instead of chaining several calls.
```js Bad theme={null}
if (!(el_exists('#price1')) || el_exists('#price2')
|| el_exists('#price3') || el_exists('#discount'))
{
dead_page('No price found');
}
```
```js Good theme={null}
if (!el_exists('#price1, #price2, #price3, #discount'))
dead_page('No price found');
```
## How do I paginate without blocking parallelization?
When a site has paginated results and you want data from every page, call `rerun_stage()` once from the root page for every page you need. Do not call `rerun_stage()` from inside each page as you walk the pagination: that serializes the work and Bright Data Scraper Studio cannot parallelize the requests.
```js Bad theme={null}
navigate(input.url);
let $ = load_html(html());
let next_page_url = $('.next_page').attr('href');
rerun_stage({url: next_page_url});
```
```js Good theme={null}
let url = new URL(input.url);
if (input.page)
url.searchParams.set('page', input.page);
navigate(url);
// input.page only exists when this stage was re-run for a specific page.
// On the root page it's undefined, so we fall through to fan out.
if (input.page)
return;
let $ = load_html(html());
let total_products = +$('.total_pages').text();
let total_pages = Math.ceil(total_products / 20);
total_pages = Math.min(total_pages, 50);
for (let page = 2; page <= total_pages; page++)
rerun_stage({url: input.url, page});
```
Avoid collecting all pages in a single session. Each session has a 16MB result size limit, and that limit counts the total accumulated data in the session, including all collected lines, parsed results, children and parser metadata, not just the size of the rendered page. Calling `rerun_stage({url: input.url, page})` once per page processes one page per session and keeps each session within the limit.
## How do I close popups without waiting for them?
Use `close_popup('popup_selector', 'close_button_selector')` to register a background watcher that closes popups whenever they appear. Do not poll for a popup with `wait_visible()` before each interaction: popups can appear at any time, and explicit waits add latency on every step.
```js Bad theme={null}
navigate('https://example.com');
try {
wait_visible('.cky-btn-accept', {timeout: 5000});
click('.cky-btn-accept');
} catch (e) {
console.log('Accept cookies button does not exist, continue');
}
```
```js Good theme={null}
// Runs in the background with no per-step latency.
// The watcher checks for the popup before every interaction automatically.
close_popup('.cky-btn-accept', '.cky-btn-accept');
navigate('https://example.com');
click('.open-product-full-info');
```
## How do I wait for a tagged response before parsing?
When you use `tag_response()` to capture a background API call, follow it with `wait_for_parser_value()` to make sure the request has finished before you read `parser`. Without the wait, the parser may run before the response has arrived and `parser.` will be `undefined`.
```js Bad theme={null}
tag_response('product', /api\/product/);
navigate('https://example.com');
// Parser code:
// The request may not have finished yet; product could be undefined
let {product} = parser;
return product.data;
```
```js Good theme={null}
tag_response('product', /api\/product/);
navigate('https://example.com');
wait_for_parser_value('product');
// Parser code:
let {product} = parser;
return product.data;
```
```js Good (chained navigation) theme={null}
tag_response('product', /api\/product/);
navigate('https://example.com');
// wait_for_parser_value returns the value so you can use it in interaction code
let product = wait_for_parser_value('product');
navigate(product.reviews_url);
tag_html('reviews_html');
// Parser code:
let {product, reviews_html} = parser;
let $ = load_html(reviews_html);
let reviews = $('.review').toArray().map(v => $(v).text());
return {
...product.data,
reviews,
};
```
## Should I throw custom error messages?
No. Let built-in errors from Bright Data Scraper Studio bubble up. They include the selector, the timeout, and the stage, which is more useful than a hand-written "Page not loaded properly". Only throw a custom error when you are checking a domain-specific condition that Bright Data Scraper Studio cannot detect on its own, such as a missing product title.
```js Bad theme={null}
try {
wait('selector1');
// some code
wait('selector2');
// some code
} catch (e) {
throw "Page not loaded properly"
}
```
```js Good theme={null}
// Crawler error: waiting for selector "selector1" failed: timeout 30000ms exceeded
wait('selector1');
// some code
wait('selector2');
// some code
```
```js Good (domain-specific check) theme={null}
if (!el_exists('.product-title'))
throw new Error('Failed to load product page');
```
## How do I handle slow websites without over-extending timeouts?
Keep the default 30-second timeout for most waits. If a specific page is consistently slow, raise it to 45 or 60 seconds. Do not push beyond 60 seconds: a slower peer is usually the cause, and Bright Data Scraper Studio automatically retries with a fresh peer session when a page reports a timeout error.
```js Bad theme={null}
// 120 seconds is too long; the platform cannot recycle a stuck peer
wait('selector', {timeout: 120000});
```
```js Good theme={null}
wait('selector'); // default 30 seconds
wait('selector', {timeout: 45000}); // 45 seconds for slightly slow pages
wait('selector', {timeout: 60000}); // 60 seconds for consistently slow pages
```
## Should I build my own retry loop?
No. Bright Data Scraper Studio handles retries at the job level with a new peer session. A custom retry loop inside your scraper reuses the same session, which is the reason the first attempt failed. Report the error and let Bright Data Scraper Studio retry.
```js Bad theme={null}
let counter = input.counter || 5;
while (counter > 1) {
try {
wait('selector', {timeout: 500});
click('selector');
type('selector');
// some code
break;
} catch (e) {
// rerun_stage creates a new session but this pattern spends extra CPM
return rerun_stage({...input, counter: --counter});
}
}
```
```js Good theme={null}
navigate('https://example.com');
wait('h1');
```
## Should I wrap parser expressions in try/catch?
No. Use optional chaining (`?.`) and nullish coalescing (`??`) instead. A silent `try/catch` around a property access hides real bugs, and a `try/catch` around a `wait()` wastes browser time.
```js Bad theme={null}
try {
const example = obj.prop;
} catch (e) {}
```
```js Bad theme={null}
// Wasting browser time for no reason
try { wait('selector'); } catch (e) {}
try { wait_network_idle({timeout: 8000}); } catch (e) {}
try { wait_page_idle(); } catch (e) {}
```
```js Good theme={null}
const example = object?.prop;
const example2 = object.prop ?? undefined;
const example3 = object.prop ? object.prop : undefined;
```
## How do I extract values from a set of elements in parser code?
Use `toArray().map()` instead of `each()`. It is shorter, returns a real array, and reads as a single expression.
```js Bad theme={null}
const links = [];
$('.card.product-wrapper').each(function (i, el) {
links.push({url: $(this).find('h4 a').attr('href')});
})
return links;
```
```js Good theme={null}
const links = $('.card.product-wrapper').toArray().map(v => ({
url: $(v).find('h4 a').attr('href'),
}));
```
## How do I normalize text in parser code?
Call `$(selector).text_sane()`. Bright Data Scraper Studio adds this custom method to the Cheerio prototype: it collapses every run of whitespace to a single space and trims the result. For numeric extraction, strip non-digits with a regex.
```js Bad theme={null}
$.prototype.clearText = function () {
return this.text().replace(/\s+/g, ' ').trim();
}
```
```js Good theme={null}
let name = $('a').text_sane();
// For digits-only extraction:
let value = +$('a').text().replace(/\D+/g, '');
```
## Related
Full reference for Bright Data Scraper Studio interaction and parser commands
Choose between Browser worker and Code worker for your scraper
# Build a scraper with the Bright Data CLI
Source: https://docs.brightdata.com/datasets/scraper-studio/build-with-the-cli
Use the Bright Data CLI to create, run and self-heal a Scraper Studio scraper from your terminal or any coding agent like Claude Code or Cursor.
The Bright Data CLI builds Bright Data Scraper Studio scrapers from your terminal in three commands: log in with `bdata login`, then `bdata scraper create` and `bdata scraper run`. Run it on demand with `npx`, so there is nothing to install. This tutorial walks you through building a Hacker News top-stories scraper end to end. The CLI runs unchanged inside the embedded terminal of any coding agent like Claude Code, Cursor or Codex.
**Time to complete:** about 10 minutes (AI generation runs in the background)
## Prerequisites
* A Bright Data account ([sign up free](https://brightdata.com/?hs_signup=1\&utm_source=docs), no card required)
* A terminal. Any embedded terminal works too: Claude Code, Cursor, Codex, VS Code
## Run the Bright Data CLI with npx
You do not install anything. Run the CLI on demand with `npx`, which fetches the latest version each time:
```bash theme={null}
npx -p @brightdata/cli bdata --version
```
To keep the rest of this tutorial's commands short, alias `bdata` for your shell session. Otherwise, prefix any `bdata` command below with `npx -p @brightdata/cli`:
```bash theme={null}
alias bdata="npx -p @brightdata/cli bdata"
```
The CLI is published as `@brightdata/cli` on npm. The `bdata` and `brightdata` commands are interchangeable. Prefer a permanent command instead of npx? Install it globally with `npm install -g @brightdata/cli`.
## Build your first scraper from the terminal
Run `bdata login`. The CLI opens a browser tab so you can authorize it against your Bright Data account, then stores your API key locally. You do not paste or copy a key.
```bash theme={null}
bdata login
```
> **Expected result:**
>
> ```text theme={null}
> Opening browser for Bright Data authentication...
> Logged in successfully. Key: 2e75****12bf
> Checking for required zones...
> Zone "cli_unlocker" already exists.
> Zone "cli_browser" already exists.
> ```
The two zones (`cli_unlocker` and `cli_browser`) are the Web Unlocker API and Browser API endpoints the CLI uses when running scrapers. Bright Data creates them automatically on first login.
Pass a target URL and one sentence describing the data you want. Bright Data's AI Agent generates the output schema, writes the scraper code and returns a Collector ID.
```bash theme={null}
bdata scraper create https://news.ycombinator.com \
"Extract top stories: title, url, points, author, comment count"
```
The AI pipeline runs in seven stages, printed live: `user_intent_analyzer`, `planner`, `collector_maintainer`, `output_schema_generator`, `code_generator`, `input_schema_generator`, `preview_runner` and `preview_picker`. Typical wall-clock time is 5 to 15 minutes; complex targets can take up to 25 minutes.
> **Expected result:**
>
> ```text theme={null}
> Template created: c_mpohus372o5tmid1jk
> Triggering AI generation...
> Generating scraper...
> Step: user_intent_analyzer — polling (attempt 1/600)
> ...
> Done in 280 poll attempts.
> {"status":"done","completed_steps":[...],"step":"preview_picker"}
> ```
Save the Collector ID (the `c_*` string). It is the stable handle for every subsequent run, schedule or API call on this scraper.
Pass the Collector ID and a URL. Use `--pretty` to format the JSON output.
```bash theme={null}
bdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --pretty
```
The CLI tries realtime mode first. If the scraper triggers more pages than the realtime limit allows, the CLI silently falls back to batch mode (`POST /dca/trigger` then poll `GET /dca/dataset`) and continues. No flag needed.
> **Expected result:** a JSON array, one row per result.
>
> ```json theme={null}
> [
> {
> "title": "Last.fm is now independent",
> "url": "https://support.last.fm/t/last-fm-is-now-independent/118591",
> "points": 447,
> "author": "twistslider",
> "comment_count": 131
> },
> {
> "title": "DuckDuckGo search saw 28% more visits after Google said people love AI mode",
> "url": "https://www.pcgamer.com/hardware/duckduckgos-ai-free-search-saw-nearly-28-percent-more-visits-in-the-week-following-googles-insistence-that-people-love-ai-mode/",
> "points": 418,
> "author": "HelloUsername",
> "comment_count": 212
> }
> ]
> ```
## How do I use this from Claude Code, Cursor or Codex?
The Bright Data CLI runs inside any embedded terminal as-is. The coding agent is not building the scraper itself; the CLI calls Bright Data's AI Agent, and the coding agent calls the CLI on your behalf.
Two integrations make the CLI feel native inside a coding agent:
**Pin the Collector ID in the agent's rules file** so the agent re-uses your scraper across sessions instead of building a fresh one every time:
```text CLAUDE.md / .cursor/rules / CODEX.md theme={null}
SCRAPER_STUDIO_COLLECTOR_ID=c_mpohus372o5tmid1jk
HACKER_NEWS_SCRAPER_USAGE="bdata scraper run $SCRAPER_STUDIO_COLLECTOR_ID --pretty"
```
**Wire Bright Data's MCP server into your agent** with `brightdata add mcp`. The MCP server is separate from the Scraper Studio CLI but gives the agent additional scraping tools (`scrape_as_markdown`, `search_engine` and others) it can call directly:
```bash theme={null}
brightdata add mcp # interactive: pick Claude Code, Cursor or Codex
```
See the [Bright Data MCP server quickstart](/ai/mcp-server-quickstart) for what the MCP exposes.
## What just happened?
Three CLI commands mapped to four Bright Data Scraper Studio API endpoints. Use this table to translate the CLI flow into raw HTTP calls when you are ready to integrate without the CLI:
| You ran | Bright Data API endpoint behind it |
| --------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `bdata login` | Local credential store. Stores the API key from [Account Settings](https://brightdata.com/cp/setting). |
| `bdata scraper create` | `POST /dca/collector` then `POST /dca/collectors/{c_*}/automate_template` |
| `bdata scraper run` (small input) | `POST /dca/trigger_immediate` then `GET /dca/get_result` |
| `bdata scraper run` (large input) | `POST /dca/trigger` then poll `GET /dca/dataset?id=j_*` |
For a worked example of the underlying API in cURL, Python and Node.js, see the [Bright Data Scraper Studio API quickstart](/datasets/scraper-studio/quickstart). For every endpoint, see the [Scraper Studio API reference](/api-reference/scraper-studio-api/Choose_a_delivery_type_on_request_level).
## How do I fix a scraper when the site changes?
When a target site is redesigned and a scraper starts returning null or missing fields, fix it in place with `bdata scraper heal`. Self-healing keeps the same Collector ID, so every trigger, schedule and integration that references the scraper keeps working. The flow is run, inspect, heal, approve, re-run.
Pass the Collector ID and a plain-language description of what broke. Keep the prompt under 1,000 characters.
```bash theme={null}
bdata scraper heal c_mpohus372o5tmid1jk \
"The points and comment_count fields return null since the site redesign. Re-capture them from the new markup." \
--url https://news.ycombinator.com
```
By default, `heal` stops at an approval gate so you can review the proposed fix before it goes live.
> **Expected result:** the command returns an envelope with `status: "awaiting_approval"` and a `preview_result` showing sample output from the proposed fix, plus a `next_step` hint with the command to run next.
Review the preview. If it looks right, approve it. The fix commits to the existing scraper and the Collector ID does not change.
```bash theme={null}
bdata scraper approve c_mpohus372o5tmid1jk --url https://news.ycombinator.com
```
To discard the proposed fix and try a sharper prompt instead, reject it:
```bash theme={null}
bdata scraper approve c_mpohus372o5tmid1jk --reject
```
> **Expected result:** on approval, `status` advances to `done`. On reject, the scraper is left unchanged so you can run `heal` again with a clearer prompt.
Run the healed scraper on the same URL and confirm the previously broken fields are populated.
```bash theme={null}
bdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com --pretty
```
For unattended workflows, add `--auto-approve` to the `heal` command. It approves the fix automatically and polls through to `done` in one step. Use it only when you trust the heal without a manual review.
## Frequently asked questions
AI generation timing depends on target complexity. Simple single-page scrapers finish in 5 to 10 minutes. Pages with lazy-load, pagination or anti-bot challenges can take 15 to 25 minutes. The CLI polls Bright Data's AI Flow API every five seconds and prints the current stage, so you can leave it running and check back. No action is needed while you wait.
Realtime mode caps the number of page loads per request. When a scraper triggers more pages than the realtime limit allows, the CLI prints `Realtime page limit exceeded, switching to batch mode...`, submits the same inputs to `POST /dca/trigger`, and polls `GET /dca/dataset?id=j_*` until the snapshot is ready. The switch is automatic and the final JSON shape is identical. See [Scraper Studio specifications](/datasets/scraper-studio/specifications) for the page-load limits.
The AI Agent's generated schema is per-row best-effort, not strict. Jobs posts, "Show HN" entries and very new submissions on Hacker News do not always have a points or comment count yet, so the scraper returns the row with those fields omitted rather than inventing a value. Treat missing fields as `null` in your own code. To enforce a stricter schema, open the scraper in [Scraper Studio](/datasets/scraper-studio/ai-agent) or rewrite the schema with the [Self-Healing tool](/datasets/scraper-studio/self-healing-tool).
Yes. The Collector ID returned by `bdata scraper create` (the `c_*` string) is the same handle the Bright Data Scraper Studio API uses. Pass it to `POST /dca/trigger` from any HTTP client. See the [Bright Data Scraper Studio API quickstart](/datasets/scraper-studio/quickstart) for cURL, Python and Node.js examples.
Fix it in place with `bdata scraper heal`, which keeps the same Collector ID. See [How do I fix a scraper when the site changes?](#how-do-i-fix-a-scraper-when-the-site-changes) above for the full run, heal, approve, re-run flow. Two alternatives:
* **Control panel:** use the [Self-Healing tool](/datasets/scraper-studio/self-healing-tool) to describe the fix in plain language.
* **Direct API:** the `heal` and `approve` commands wrap a three-call loop. `POST /dca/collectors/{c_*}/refactor_template` with the prompt, poll `GET .../refactor_template/progress` until `status` is `pending_answer`, then `POST .../resume_automation_job` to approve or reject. See [Trigger Self-Healing](/api-reference/scraper-studio-api/ai-flow/trigger-self-healing) and [Resume Self-Healing Job](/api-reference/scraper-studio-api/ai-flow/resume-self-healing-job). For a Node.js implementation, see the [Scraper Studio Self-Healing demo](https://github.com/anil-bd/scraper-studio-self-healing-demo).
The `bdata login` command requires a browser callback. For headless environments, export your API key as `BRIGHTDATA_API_KEY` and the CLI uses it directly without a login step:
```bash theme={null}
export BRIGHTDATA_API_KEY="your_api_key_here"
bdata scraper run c_mpohus372o5tmid1jk https://news.ycombinator.com
```
Copy the key from [Account Settings](https://brightdata.com/cp/setting).
## Related
Build the same scraper from the Bright Data control panel instead of the terminal
Trigger an existing scraper from cURL, Python or Node.js
Fix a scraper with a plain-language prompt when a target site changes
Every `bdata` command, with examples
# Scraper Studio coding agent prompts
Source: https://docs.brightdata.com/datasets/scraper-studio/coding-agent-prompts
Copy-pasteable Claude Code, Cursor and Codex prompts to build a Bright Data Scraper Studio scraper: a simple build-and-run prompt plus a full self-heal loop.
Use these copy-pasteable prompts to build a Bright Data Scraper Studio scraper through Claude Code, Cursor or Codex. Start with the one-prompt build-and-run if you just want a scraper fast: swap in your URL and the fields you want, paste, done. The longer flows below add the full build, run, heal, approve, re-run loop for when you need to extend a scraper's schema in place.
## Prerequisites
* A Bright Data account ([sign up free](https://brightdata.com/?hs_signup=1\&utm_source=docs), no card required).
* A coding agent with terminal access: Claude Code, Cursor or Codex.
You do not install the Bright Data CLI ahead of time. The prompts below run it through `npx`, which fetches the latest version on demand, so there is no global dependency to maintain.
## Build a scraper in one prompt
If you just want a scraper fast, paste this prompt and replace the two values in angle brackets: the target URL and the fields you want. The agent runs the Bright Data CLI through `npx`, builds the scraper and runs it once. No self-healing.
```text Prompt theme={null}
Build and run a Bright Data scraper. Run every Bright Data CLI command through `npx -p @brightdata/cli` so nothing is installed globally. Replace and , then do each step in order and stop if a step fails:
1. Authenticate by running `npx -p @brightdata/cli bdata login`. npx fetches the CLI on demand, so there is nothing to install.
2. Create a Bright Data scraper for that extracts: . Report the Collector ID.
3. Run that scraper on the same URL and pretty-print the result.
```
For example, to scrape a product page, the filled-in second step reads:
```text Prompt theme={null}
2. Create a Bright Data scraper for https://shopalto.xyz/product/aurora-wireless-headphones that extracts: product name, price, description and rating. Report the Collector ID.
```
> **Expected result:** the agent reports a Collector ID like `c_mpohus372o5tmid1jk`, then prints a JSON array with one row containing the fields you asked for.
Save the Collector ID. Reuse it to run the scraper on new URLs, or to extend its schema later with the self-heal flow below.
## Build, run and self-heal in one prompt
To run the full build, run, heal, approve, re-run loop in action, paste this single prompt and let the agent work through every step. The pattern is deliberate: build a minimal scraper first, then heal it to extend the schema, so the heal envelope's `preview_result` is easier to verify against a known-good baseline. The agent runs every Bright Data CLI command through `npx`, so nothing is installed globally.
```text Prompt theme={null}
Build, run, heal and verify a Bright Data scraper end to end. Run every Bright Data CLI command through `npx -p @brightdata/cli` so nothing is installed globally. Do every step in order and stop if a step fails:
1. Authenticate by running `npx -p @brightdata/cli bdata login`. npx fetches the CLI on demand, so there is nothing to install.
2. Create a Bright Data scraper for https://shopalto.xyz/product/aurora-wireless-headphones that extracts two fields: product name and price. Report the Collector ID.
3. Run that scraper on the same URL and pretty-print the result. Expect one row with name and price.
4. Heal the scraper in place to also capture description, image url and rating alongside the existing name and price. Keep the same Collector ID, anchor the heal on the same URL and show the approval envelope.
5. When the preview shows all five fields, approve the fix anchored on the same URL.
6. Run the scraper on the same URL again and confirm all five fields come back: name, price, description, image_url and rating.
```
> **Expected result:** the agent ends with a JSON row containing `name`, `price`, `description`, `image_url` and `rating`, and the Collector ID is unchanged from step 2.
## Run the flow step by step
Work through the prompts below one at a time when you want to inspect each Collector ID, run result and heal envelope before moving on.
```text Prompt theme={null}
Run every Bright Data CLI command through `npx -p @brightdata/cli` so nothing is installed globally. Authenticate by running `npx -p @brightdata/cli bdata login`, then confirm the version with `npx -p @brightdata/cli bdata --version` before continuing.
```
> **Expected result:** the agent prints a `bdata` version and confirms it is authenticated. npx fetches the CLI on demand, so nothing is installed globally.
```text Prompt theme={null}
Create a Bright Data scraper for https://shopalto.xyz/product/aurora-wireless-headphones that extracts just two fields: product name and price. Show me the Collector ID when it is done.
```
> **Expected result:** the agent reports a Collector ID like `c_mpohus372o5tmid1jk`. Hold onto it; the rest of the prompts reuse the same ID.
```text Prompt theme={null}
Run that scraper on https://shopalto.xyz/product/aurora-wireless-headphones and pretty-print the result.
```
> **Expected result:** a JSON array with one row, populated with `name` and `price` only.
```text Prompt theme={null}
Extend the scraper in place. Heal it to also capture description, image url and rating alongside the existing name and price. Keep the same Collector ID. Anchor the heal on https://shopalto.xyz/product/aurora-wireless-headphones and show me the approval envelope when it is ready.
```
> **Expected result:** the agent reports `status: "awaiting_approval"` with a `preview_result` row that now shows five fields.
```text Prompt theme={null}
The preview looks good. Approve the fix, anchored on https://shopalto.xyz/product/aurora-wireless-headphones.
```
> **Expected result:** `status` advances to `done`. The Collector ID is unchanged.
```text Prompt theme={null}
Run the scraper on https://shopalto.xyz/product/aurora-wireless-headphones again and confirm all five fields now come back: name, price, description, image_url and rating.
```
> **Expected result:** the same JSON shape as the earlier run, now with three additional fields per row.
For an unattended variant, ask the agent to add `--auto-approve` to the heal call. The agent skips the approval gate and polls through to `done` in one step. Use it only when you trust the heal without a manual review.
## Related
The canonical CLI tutorial: install, log in, create, run, heal
Fix a scraper from the Bright Data control panel
Trigger an existing scraper from cURL, Python or Node.js
Flag reference for scraper create, heal and approve
# Complete Scraper Studio IDE examples
Source: https://docs.brightdata.com/datasets/scraper-studio/complete-examples
Browse 6 end-to-end Bright Data Scraper Studio IDE examples covering interaction code, parser code, multi-result handling and advanced site navigation.
This article applies only to the IDE Normal V2 version, which is for internal use and not available to customers.
## Introduction
The collect and parse commands have been removed. The data will be returned from parser code as an object or array, and it will be automatically saved to the output:
```js Interaction code theme={null}
// Old code
navigate("https://example.com");
collect(parse());
// New code
navigate("https://example.com");
// New code alternative
navigate("https://example.com");
tag_html("html_key");
```
```js Parser code theme={null}
// Old code
return {
title: $('h1').text(),
};
// New code
return {
title: $('h1').text(),
};
// or
let $ = load_html(parser.html_key);
return {
title: $('h1').text(),
};
```
New commands have been added to provide access to the data from the interaction code: `tag_html`, `tag_request`, `tag_graphql`
In addition, some existing commands have been updated: `tag_response`, `tag_sitemap`, `tag_all_responses`. See [IDE documentation](/datasets/scraper-studio/develop-a-scraper) for more details.
When using any tag commands, you can provide a custom name. You can then access the data using this name in the parser code under parser.`YOUR_KEY`.
For `tag_html`, current browser location URL will be saved under parser.`YOUR_KEY_url`.
For simple cases when only a single `tag_html` is needed, it can be skipped, and it will be automatically saved under parser.page.
```js Interaction code theme={null}
navigate("https://example.com/1");
tag_html("page1");
navigate("https://example.com/2");
tag_html("page2");
navigate("https://example.com/3");
tag_html("page3");
```
```js Parser code theme={null}
{page1, page1_url, page2, page2_url, page3, page3_url} = parser;
let $ = load_html(page2);
return {
title2: $('h1').text(),
};
```
Sometimes, it's necessary to get parsed data within the interaction code and use it to make request. See examples how to do it:
```js Interaction code theme={null}
navigate("https://example.com/1");
tag_html("page1");
let page_html = html();
let page_html2 = wait_for_parser_value("page1"); // the same
let $ = load_html(page_html);
let req_id = $('.product-id').text();
tag_request("product_json", {url: "https://example.com/product/"+req_id});
```
```js Parser code theme={null}
{product_json} = parser;
return product_json;
```
## How to return multiple results
To collect multiple results, array can be returned from the parser code.
```js Interaction code theme={null}
navigate("https://example.com/products");
```
```js Parser code theme={null}
let items = $(".product").toArray().map(v=>$(v)).map(v=>({
title: v.find(".title").text(),
price: v.find(".price").text(),
url: new URL(v.find("a").attr("href")),
}));
return items;
```
## How to reparse collected data
Reparse is a new feature that allows to reparse the data that was already collected. It can be useful when you want to change the parser code without rerunning the entire interaction code:
## `next_stage` and `rerun_stage`
When scraper has more than one step, the parser code is only available in the last step. All other steps can only have `next_stage` calls. To parse something from the page, `load_html` should be used:
```js Interaction code step 1 theme={null}
navigate(input.domain_url);
tag_html('html');
const $ = load_html(wait_for_parser_value('html'));
$('a.layout-categories-category__link').toArray()
.map(v => new URL($(v).attr('href'), location.href))
.filter(x => !x?.href.includes('home')).forEach(i => {
next_stage({url: i})
});
```
```js Interaction code step 2 theme={null}
block(['*.ico', '*.png', '*.jpg', '/images/', '*gif']);
navigate(input.url);
wait('.view-option-selector-button, .product-groups_empty-list');
if(el_exists('.product-groups_empty-list'))
dead_page('There are no products in this category')
wait('[aria-describedby="onetrust-policy-text"]');
click('button#onetrust-reject-all-handler');
wait_hidden('[aria-describedby="onetrust-policy-text"]');
$('.view-option-selector-button').eq(2).click();
wait_page_idle(2000);
scroll_to('bottom');
wait_page_idle(2000);
```
```js Parser code theme={null}
return $('a[data-qa-action="product-click"]').toArray()
.map(v => ({url: new URL($(v).attr('href'))}));
```
## Basic PDP scraper
```js Interaction code theme={null}
let url = new URL(input.url.replace('https://www.slintel.com','https://6sense.com'));
url = new URL(url.pathname, 'https://6sense.com');
navigate(url);
if (location.href === 'https://6sense.com/company')
dead_page(`Page not found`);
tag_html('html');
```
```js Parser Code theme={null}
const $ = load_html(parser.html)
const nextData = JSON.parse($('#__NEXT_DATA__').html());
const pageProps = nextData.props.pageProps;
const companyInfo = pageProps.company_data.companyInfo;
const linkedin = companyInfo.linkedin ?
(companyInfo.linkedin.includes('http') ? companyInfo.linkedin :
'https://'+companyInfo.linkedin) : null;
const techCategories = pageProps.company_data?.technologies_mapper_view?.categories
? Object.values(pageProps.company_data.technologies_mapper_view.categories).map(v => Object.keys(v)).flat()
: null;
const pageData = {
"name": companyInfo.name,
"about": companyInfo.company_description,
"num_employees": companyInfo.employee_range,
"type": companyInfo.company_type,
"industries": companyInfo.industry_v2_ranked.filter(v=>v),
"techstack_arr": techCategories,
"country_code": companyInfo.country,
"website": companyInfo.display_domain
? new URL(companyInfo.display_domain.includes('http')
? companyInfo.display_domain
: 'https://'+companyInfo.display_domain)
: null,
"social_media_urls": linkedin,
"company_news": pageProps.company_data.company_news.map(v=>({
title: "company_news_data",
date: "",
link: "",
})),
"last_updated": new Date(companyInfo.last_updated_at*1e3),
"url": new URL(parser.html_url),
"logo": companyInfo.logo,
"location": companyInfo.location,
"region": [companyInfo.country, companyInfo.state].filter(v=>v).join(', '),
"id": nextData.query.companyid,
"slintel_resources": companyInfo.recommended_companies.map(v=>({
link: v.display_domain ? new URL(v.display_domain.includes('http') ? v.display_domain : 'https://'+v.display_domain) : null,
title: v.name,
type: v.company_type,
})),
"stock_symbol": companyInfo.stock_symbol,
};
return pageData;
```
## Multiple navigates example
```js Interaction code theme={null}
const tabs = [
'topactivity',
'answers',
'questions',
'tags',
'articles',
'badges',
'bookmarks',
'bounties'
];
function loadTab(url, name) {
return new Promise(async (resolve, reject)=>{
try {
const tabUrl = new URL(url);
tabUrl.searchParams.set('tab', name);
navigate(tabUrl, { allow_status: [404] });
const html_ = html();
tag_html(name);
resolve(html_);
} catch(e) { reject(e); }
});
}
const userIdFromUrl = (input.url && input.url.includes('https://stackoverflow.com/users/'))
? input.url.replace('https://stackoverflow.com/users/', '').split('/').shift()
: null;
const userId = input.user_id || userIdFromUrl || 1;
const userUrl = `https://stackoverflow.com/users/${userId}`;
navigate(userUrl, { allow_status: [404] });
tag_html('user');
Promise.all(tabs.map( async tabName => loadTab(userUrl, tabName)));
```
```js Parser code theme={null}
const steps = [
'user',
'topactivity',
'answers',
'questions',
'tags',
'articles',
'badges',
'bookmarks',
'bounties',
'finally'
];
const parse = Handlers();
let missed_parser_data = Object.keys(parse)
.filter(handler_key=>!steps.find(sk=> sk == handler_key));
if (missed_parser_data.length)
throw new Error('missed parser data')
return steps.reduce((acc, step) => {
console.log('handling: '+step)
if (step=='finally')
return parse.finally(acc);
let page_source = parser[step];
console.log(parser[step]);
console.log('parser[step]');
if (!page_source)
throw('unexpected empty data, data not saved')
const $ = load_html(page_source);
const data_chunk = parse[step]($);
acc = { ...acc, ...data_chunk };
return acc;
}, {});
function Handlers() {
return {
'finally': (res) =>{
res.url = new URL(res.url);
res.answers.map(v => v.tags = res.tags);
res.answers.map(v => v.url = new URL(v.url));
res.questions.map(v => v.url = new URL(v.url));
res.top_posts.map(v => v.date = new Date(v.date));
res.active_from = new Date(res.active_from)
return res;
},
'user': ($)=>{
const grid = $('#main-content .d-grid > .grid--item')
.toArray().map(v=>$(v));
let collectives = grid.find(v=>v.find('> div').text()
.includes('Collectives'));
if (collectives?.length) {
collectives = collectives.find('.fl1').toArray()
.map(v=>({
name: $(v).find('.fs-body2').text_sane() || null,
text: $(v).find('.fs-caption').text_sane() || null,
}));
}
else
collectives = [];
let communities = grid.find(v=>v.find('> div').text()
.includes('Communities'));
if (communities?.length) {
communities = communities.find('li.flex--item')
.toArray().map(v=>({
name: $(v).find('.fl-grow1').text_sane() || null,
score: $(v).find('.fl-shrink0').text_sane()
.replace(/,/gm, '') || null,
}));
}
else
communities = [];
let badges = $('.flex__fl-equal > .flex--item').toArray()
.map(v=>{
let $v = $(v);
return {
badge_type: $v.find('.mr12 .fc-gold').length
? 'gold'
: $v.find('.mr12 .fc-silver').length
? 'silver'
: $v.find('.mr12 .fc-bronze').length
? 'bronze'
: 'unknown',
badge_name: $v.find('.fs-caption').text_sane()
.replace(' badges', '') || null,
number_of_badges: $v.find('.fs-title').text()
.replace(/,/gm, '') || null,
badges: $v.find('a.badge').toArray().map(b=>({
name: $(b).text_sane() || null,
description: $(b).attr('title') || null
}))
};
});
return {
url: new URL(parser.user_url),
id: parser.user_url.split('/').pop(),
'id#': (input.url && input.url
.includes('https://stackoverflow.com/users/'))
? input.url?.match(/https:\/\/stackoverflow.com/users/(d{1,15}).*/)?.[1]/gm)
: input.user_id
|| 1,
user_id: +$('[property="og:url"]').attr('content')
.match(/https:\/\/stackoverflow.com/users/(d{1,15}).*/)?.[1]/)/gm
|| +input.url?.match(/https:\/\/stackoverflow.com/users/(d{1,15}).*/)?.[1]/gm)
|| +input.user_id
|| null,
name: $('.fs-headline2').text_sane() || null,
type: $('#mainbar-full > div:first-child .s-badge')
.text_sane() || null,
title: $('#mainbar-full > div:first-child .fs-title')
.text_sane() || null,
active_from: new Date(
$('#mainbar-full > div:first-child .list-reset span[title]').attr('title')),
last_seen: $('#mainbar-full > div:first-child .fs-title + ul li:last-child')
.text_sane() || null,
linkes: $('#mainbar-full > div:first-child ul.list-reset:last-child a')
.toArray().map(v => $(v).attr('href')),
location: $('#mainbar-full > div:first-child ul.list-reset:last-child li:last-child div[title]')
.attr('title') || null,
stats: $('#stats .flex--item').toArray()
.map(v => $(v).text_sane()).filter(v => v),
about: $('#main-content .s-prose p').toArray()
.map(v => $(v).text_sane()) || null, //.join('\r'),
collectives,
communities,
badges,
top_tags: $('#top-tags .p12').toArray().map(v=>({
name: $(v).find('.s-tag').text_sane() || null,
score: $(v).find('.d-flex > .d-flex:nth-child(1)')
.text_sane().replace(' Score', '')
.replace(/,/gm, '') || null,
posts: $(v).find('.d-flex > .d-flex:nth-child(2)')
.text_sane().replace(' Posts', '')
.replace(/,/gm, '') || null,
posts_percent: $(v).find('.d-flex > .d-flex:nth-child(3)')
.text_sane().replace(' Posts %', '').replace(/,/gm, ''),
})),
top_posts: $('#js-top-posts .p12 .d-flex').toArray().map(v=>({
type: $(v).find('.iconQuestion').length ? 'question'
: ($(v).find('.iconAnswer').length ? 'answer' : 'none'),
score: $(v).find('.s-badge').text_sane()
.replace(/,/gm, '') || null,
text: $(v).find('a.d-table')
.text_sane() || null,
date: new Date($(v).find('.relativetime')
.attr('title')),
})),
top_meta_posts: $('#js-top-posts + div .p12 .d-flex')
.toArray().map(v=>({
score: $(v).find('.s-badge').text_sane()
.replace(/,/gm, '') || null,
text: $(v).find('a.d-table').text_sane() || null,
})),
top_network_posts: $('#js-top-posts + div + div .p12 .d-flex')
.toArray().map(v=>({
score: $(v).find('.s-badge').text_sane()
.replace(/,/gm, '') || null,
text: $(v).find('a.d-table').text_sane() || null,
}))
};
},
'topactivity': ($) => {
let summary_graph_data;
try {
summary_graph_data = JSON.parse(
/graphDatas[^[]+([[^]]+])/gm.exec(
$('*').first().html())?.[1]);
} catch(e) {
console.log('graphData not found')
}
let [summary_people_reached, summary_posts_edited,
summary_helpful_flags, summary_votes_cast] =
$('div:not(.js-highlight-box-reputation) > h1.flex--item + div .flex--item .fs-body3')
.toArray().map(v=>$(v).text_sane().replace(/,/gm, '') || null);
return {
summary_reputation: +$('#top-cards h4.fs-headline1')
.text().replace(/D+/gm, ''),
summary_top_overall: $('a[href*="alltime"]')
.first().text_sane() || null,
summary_next_tag_badge: $('#rep-card-next-tag-badge a')
.text_sane() || null,
summary_graph_data,
summary_next_tag_score:
$('div.fl-shrink1 + div.fl-grow1 .fs-fine')
.first().text_sane().replace(/,/gm, '') || null,
summary_next_tag_answers:
$('div.fl-shrink1 + div.fl-grow1 .fs-fine')
.last().text_sane().replace(/,/gm, '') || null,
summary_badges: $('h3 +div .s-badge').toArray()
.map(v => $(v).attr('title')).join(', '),
summary_last_badge: $('#badge-card-last').text_sane() || null,
summary_next_badge: {
name: $('#js-badge-card-next').text_sane() || null,
progress: $('h4.flex--item.ws-nowrap + span')
.text_sane() || null
},
summary_people_reached,
summary_posts_edited,
summary_helpful_flags,
summary_votes_cast,
};
},
'answers': ($) => {
return {
answers: $('#js-post-summaries .s-post-summary')
.toArray().map(p=>{
let $p = $(p);
return {
id: $p.attr('data-post-id') || null,
votes: $p.find('.s-post-summary--stats-item-number')
.text().replace(/,/gm, '') || null,
accepted: !!$p.find('svg.iconCheckmarkSm').length,
url: new URL($p.find('.answer-hyperlink')
.attr('href'), location.href),
text: $p.find('.answer-hyperlink').text_sane() || null,
tags: $p.find('.post-tag').toArray()
.map(v => $(v).text_sane() || null ).map(name=>({name}))
};
}),
}
},
'questions': ($) => {
return {
questions: $('#js-post-summaries .s-post-summary')
.toArray().map(p=>{
let $p = $(p);
return {
id: $p.attr('data-post-id') || null,
votes: $p.find('.s-post-summary--stats-item-number')
.text().replace(/,/gm, '') || null,
accepted: !!$p.find('svg.iconCheckmarkSm').length,
answer_count:
$p.find('.s-post-summary--stats-item:nth-child(2) .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
url: new URL($p.find('.s-link').attr('href'),
location.href),
text: $p.find('.s-link').text_sane() || null,
tags: $p.find('.post-tag').toArray()
.map(v => $(v).text_sane() || null)
};
}),
};
},
'tags': ($) => {
return {
tags: $('#user-tab-tags .ba .p12').toArray()
.map(v=>$(v)).map(v=>({
name: v.find('.post-tag').text_sane() || null,
badge: v.find('.badge-tag').attr('title') || null,
description: v.attr('title') || null,
score: v.find('.flex--item:first-child > .fs-body3')
.text_sane().replace(/,/gm, '') || null,
posts: v.find('.flex--item:last-child > .fs-body3')
.text_sane().replace(/,/gm, '') || null,
})),
};
},
'articles': ($) => {
return {
articles_count: $('h2.fs-title').text()
.replace(/D+/gm, '') || null
};
},
'badges': ($) => {
return {
badges: $('#user-tab-badges .grid--item').toArray()
.map(v => ({
name: $(v).find('.mbn4').text_sane() || null,
count: $(v).find('.ml4').text_sane()
.replace(/D+/gm, '') || 0
})),
};
},
'bookmarks': ($) => {
return {
bookmarks: $('#js-post-summaries .s-post-summary')
.toArray().map(p=>{
let $p = $(p);
return {
id: $p.attr('data-post-id'),
votes:
$p.find('.s-post-summary--stats-item__emphasized .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
views: $p.find('.is-supernova .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
accepted: !!$p.find('svg.iconCheckmarkSm').length,
answer_count: $p.find('.s-post-summary--stats-item:nth-child(2) .s-post-summary--stats-item-number')
.text_sane().replace(/,/gm, '') || null,
url: new URL($p.find('.s-link').attr('href'),
location.href).href,
text: $p.find('.s-link').text_sane() || null,
tags: $p.find('.post-tag').toArray()
.map(v => $(v).text_sane() || null)
};
})
};
},
'bounties': ($) => {
return {
bounties_count: $('h2.fs-title').text()
.replace(/D+/gm, '') || null
};
},
}
}
```
## Multiple `tag_response`
```js Interaction code theme={null}
close_popup('._1piuevz', '._1piuevz');
tag_response('stay', /\/StaysPdpSections/);
tag_response('calendar', /\/PdpAvailabilityCalendar/);
tag_response('reviews', /api\/v3\/StaysPdpReviews/);
navigate(input.url+'?enable_auto_translate=false#availability-calendar');
if (el_exists('img[src*="error_pages/404"]'))
return dead_page('Page not found');
const room_id = /\/(\d+)(?:\?|$)/.exec(input.url)?.[1];
if (el_exists('._wgmchy ._1qx9l5ba'))
click('._wgmchy ._1qx9l5ba');
wait('#data-state, #data-deferred-state',{timeout: 6e4});
wait_for_parser_value('stay');
wait_for_parser_value('calendar');
wait_for_parser_value('reviews');
tag_html('html');
```
```js Parser code theme={null}
let room_id = /([0-9]+)(?:\?|$)/.exec(input.url)?.[1];
const $ = load_html(parser.html);
let price = +$('[data-section-id="BOOK_IT_SIDEBAR"] ._14y1gc ._tyxjp1')
.first().text().replace(/[^[0-9].,]+/gm,'')
let currency = $('._tyxjp1').last().text().replace(/[0-9,]/g, '').trim()
let data = JSON.parse($('#data-state, #data-deferred-state').html())
price = price ? new Money(price, currency) : null
let avatar = $('._9ofhsl').attr('src')
let name = $('.tehcqxo h2.hnwb2pb').text().split('by')?.[1]?.trim()
|| $('._cv5qq4 ._14i3z6h').text().split('by')?.[1]?.trim()
let { stay, calendar, reviews} = parser;
let cd = data.niobeMinimalClientData.find(f=>f.find(f2=>
f2?.data?.presentation?.stayProductDetailPage))?.find(f2=>
f2?.data?.presentation?.stayProductDetailPage)
.data?.presentation?.stayProductDetailPage?.sections;
let sections = stay.data.presentation.stayProductDetailPage.sections
.sections || [];
sections = sections.concat(cd?.sections);
function get_section(name) {
return sections.find(v=>v.sectionId == name).section;
}
let metadata = stay.data.presentation.stayProductDetailPage.sections
.metadata || cd?.metadata;
let category_rating = [];
sections.find(v=>v?.sectionId == "REVIEWS_DEFAULT")?.section
?.ratings?.forEach(r=>{
category_rating.push({
name: r.label,
value: r.localizedRating,
});
});
let sec_sheet = get_section("BOOK_IT_CALENDAR_SHEET");
let sec_book_it = get_section("BOOK_IT_SIDEBAR");
price = sec_book_it.structuredDisplayPrice.price
?.replace(/[^[0-9].]/,'') || price;
currency = data.userAttributes?.curr || currency;
let available_dates = [];
calendar.data.merlin.pdpAvailabilityCalendar.calendarMonths.forEach(m=>{
m.days.forEach(d=>{
if (d.available) available_dates.push(d.calendarDate);
});
});
let image = metadata.sharingConfig.imageUrl ||
sections.find(v=>v.sectionId == "PHOTO_TOUR_SCROLLABLE_MODAL")
.section.mediaItems[0].baseUrl;
if (image) image = new Image(image);
avatar = sections.find(v=>v?.sectionId == "HOST_PROFILE_DEFAULT")
?.section?.hostAvatar?.avatarImage?.baseUrl || avatar;
if (avatar) avatar = new Image(avatar);
return {
name: metadata.sharingConfig.title,
price,
image,
description: sections.find(v=>v?.sectionId == "DESCRIPTION_MODAL")
?.section?.items?.map(v=>((v?.title||'')+' '+v?.html?.htmlText).trim()
)?.join(', '),
category: metadata.seoFeatures.breadcrumbDetails[1].linkText,
availability: sec_sheet.available.toString(),
reviews: reviews.data.presentation.stayProductDetailPage.reviews
.reviews.map(r=>r.localizedReview?.comments || r.comments),
ratings: metadata.sharingConfig.starRating,
seller_info: {
name: sections.find(v=>v?.sectionId == "HOST_PROFILE_DEFAULT")
?.section?.title || name,
url: new URL(`https://www.airbnb.ru/contact_host/${room_id}/send_message`),
avatar,
features: sections.find(v=>v?.sectionId == "HOST_PROFILE_DEFAULT")
?.section?.hostFeatures?.map(f=>({
name: f.title,
value: f.subtitle
}))
},
breadcrumbs: sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")
?.section?.previewLocationDetails?.[0]?.title,
location: sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")
?.section?.previewLocationDetails?.[0]?.title,
lat: metadata.loggingContext.eventDataLogging.listingLat ||
sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")?.section?.lat,
long: metadata.loggingContext.eventDataLogging.listingLng ||
sections.find(v=>v?.sectionId == "LOCATION_DEFAULT")?.section?.long,
guests: sec_book_it.maxGuestCapacity || sections
.find(v=>v?.sectionId == "AVAILABILITY_CALENDAR_DEFAULT")
?.section?.maxGuestCapacity,
pets_allowed: sec_book_it.petsAllowed,
description_items: sections.find(v=>
v?.sectionId == "AVAILABILITY_CALENDAR_DEFAULT")?.section
?.descriptionItems.map(v=>v.title),
category_rating,
house_rules: sections.find(v=>v?.sectionId == "POLICIES_DEFAULT")
?.section?.houseRules?.map(r=>r?.title),
details: sections.find(v=>v?.sectionId == "OVERVIEW_DEFAULT")
?.section?.detailItems?.map(f=>f?.title),
highlights: sections.find(v=>v?.sectionId == "HIGHLIGHTS_DEFAULT")
?.section?.highlights?.map(f=>({
name: f.title,
value: f.subtitle
})),
arrangement_details: sections.find(v=>
v?.sectionId == "SLEEPING_ARRANGEMENT_DEFAULT")?.section
?.arrangementDetails?.map(f=>({
name: f.title,
value: f.subtitle
})),
amenities: sections.find(v=>v?.sectionId == "AMENITIES_DEFAULT")
?.section?.seeAllAmenitiesGroups?.map(g=>({
group_name: g.title,
items: g.amenities.filter(f=>f.available).map(f=>({
name: f.title,
value: f.subtitle
}))
})),
images: sections.find(v=>v?.sectionId == "HERO_DEFAULT")
?.section?.previewImages?.map(v=>new Image(v?.baseUrl)),
available_dates,
url: new URL(input.url),
final_url: new URL(location.href),
};
```
# Develop a scraper with the IDE
Source: https://docs.brightdata.com/datasets/scraper-studio/develop-a-scraper
Build a custom web scraper in the Bright Data Scraper Studio IDE: write interaction and parser code, preview results, debug runs, and ship to production.
This guide walks through building a custom web scraper in the Bright Data Scraper Studio IDE from scratch. You will write interaction code that navigates the target site, parser code that extracts structured fields, and then save the scraper to production and configure delivery. By the end, you will have a runnable scraper you can trigger by API, manually, or on a schedule.
## Prerequisites
* An active [Bright Data account](https://brightdata.com/?hs_signup=1\&utm_source=docs) with access to Scraper Studio
* Basic JavaScript familiarity (variables, functions, control flow)
* A target URL you want to scrape
If you prefer describing the scraper in plain language instead of writing code, use the [Scraper Studio AI Agent](/datasets/scraper-studio/ai-agent). The agent generates the same kind of scraper the IDE would produce.
## How do I build a scraper in the IDE?
Go to [brightdata.com/cp/scrapers](https://brightdata.com/cp/scrapers), click **Scraper Studio**, then click **Develop a web scraper (IDE)** to open an empty scraper.
You can start from a blank scraper or use a template.
Templates are pre-built scraper starters for common patterns and sites. Use a template when your target site or scraping pattern is similar to one of the available options.
Use a blank scraper when you want full control over the scraping flow from the beginning.
Interaction code navigates the target site and captures the data you need into the parser. Select **Interaction code** from the left sidebar and write the code in the main editor.
A minimal interaction script:
```js theme={null}
navigate(input.url);
wait('.product-title');
let data = parse();
collect(data);
```
In this example:
* `navigate(input.url)` opens the page provided in the input.
* `wait('.product-title')` waits for the expected element.
* `parse()` runs the parser code.
* `collect(data)` adds the parsed record to the output dataset.
For a multi-page scrape, fan out with `next_stage()`:
```js theme={null}
navigate(input.url);
wait('.listing');
let listings = parse().listings;
for (let url of listings)
next_stage({url});
```
See [Scraper Studio functions](/datasets/scraper-studio/functions) for the full list of interaction commands.
Parser code reads the HTML or response loaded by interaction code and returns a structured JavaScript object.
Select **Parser code** from the left sidebar and define the fields you want to extract.
Parser code commonly uses Cheerio’s jQuery-like `$` selector:
```js theme={null}
return {
title: $('h1').text_sane(),
price: new Money(+$('.price').text().replace(/\D+/g, ''), 'USD'),
image: new Image($('img.product').attr('src')),
listings: $('.listing a').toArray().map(el => $(el).attr('href')),
};
```
The object returned by parser code is available wherever interaction code calls `parse()`. See [Scraper Studio functions](/datasets/scraper-studio/functions#parser-functions) for the parser helpers Bright Data Scraper Studio provides.
In the **Settings** panel, pick the worker type:
* **Code worker** (faster): for static HTML pages and public JSON endpoints
* **Browser worker**: for JavaScript-rendered pages, clicks, scrolling, popups, or captured background traffic
Start with **Code worker** when possible. Switch to **Browser worker** if the data you need is not available in the raw response or if you need [browser-only](/datasets/scraper-studio/functions#browser-only-functions) functions.
See [**Worker types**](/datasets/scraper-studio/worker-types) for the full comparison.
Use **Preview** to test the scraper before making it active for production use.
Preview runs the scraper code against the input selected in the **Input** tab at the bottom-left of the IDE. Use it to test interaction logic, parser logic, output structure, and errors before running the scraper on a larger input set.
The results appear in the **Output** tab. Use the **Run log** and **Browser network** tabs to debug failed runs.
> **Expected result:** the Output tab shows a structured record with the fields defined in your parser code.
When the preview returns the expected output, click **Finish editing** in the top-right corner For an existing production scraper, click **Save to production** to apply your changes.
The scraper appears under **My Scrapers** in the control panel and can be triggered by API, manually, or on a schedule.
Open the scraper from **My Scrapers** and configure **Delivery preferences**. Choose a delivery destination, such as API download, webhook, Amazon S3, GCS, Azure, SFTP, or email and a file format (JSON, NDJSON, CSV, XLSX). See [Initiate collection and delivery](/datasets/scraper-studio/initiate-collection-and-delivery-options) for all available options.
After the scraper is active and delivery is configured, start a production run. Choose the initiation method that matches your workflow:
* [Initiate by API](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run) - start a run from your application or automation workflow.
* [Initiate manually](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run) - start a run from the control panel by entering inputs or uploading a file.
* [Schedule a scraper](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run) - run the scraper automatically on a recurring schedule.
## Frequently asked questions
Open the scraper in the Bright Data Scraper Studio IDE and check the **Last errors** tab. Every failed input is stored with its exact error message and error code (up to the most recent 1,000 failures). Re-run the failing input from the IDE to reproduce the problem locally, fix the interaction or parser code, and save a new production version.
Yes. Every scraper in Bright Data Scraper Studio, regardless of how it was created, can be opened and edited in the IDE. You can change extraction logic, tweak selectors, add or remove output fields, and change the worker type.
Click **Edit Schema** in the IDE's output schema panel and add the new fields, or return them from parser code and Bright Data Scraper Studio prompts you to update the schema when you save to production.
Use `collect()` to append one record at a time; it is the default way to emit data. Use `set_lines()` when you are collecting records progressively and want the most recent snapshot delivered even if a later step throws an error. Every call to `set_lines()` overrides the previous one. See [collect](/datasets/scraper-studio/functions#collect-append-a-record-to-the-dataset) and [set\_lines](/datasets/scraper-studio/functions#set-lines-set-output-lines-overriding-previous-calls).
## Related
Full reference for interaction and parser commands
Recommended patterns for fast, reliable scrapers
Reference for every panel and control in the IDE
Fix broken scrapers and add fields with plain-language prompts
# Scraper Studio error codes
Source: https://docs.brightdata.com/datasets/scraper-studio/error-codes
Look up Bright Data Scraper Studio error codes, status codes and warnings, with the meaning and suggested fix for each collection failure.
This reference lists the error codes Bright Data Scraper Studio returns when a crawl, parser, request, validation or delivery step fails during a collection run, and the suggested action for each one.
Read the `error_code`, `status_code` and raw `error` message together to understand what failed and what to do next.
These are Scraper Studio collection errors, not API authentication or API request errors. For proxy-layer HTTP errors returned outside Scraper Studio, see the [Error catalog](/proxy-networks/errorCatalog).
## How do I read a Scraper Studio error?
Scraper Studio uses five system fields to describe the status of each record.
| Field | Type | What it tells you |
| -------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `error_code` | Machine-readable identifier | Why the crawl failed, for example `dead_page`, `bad_input`, `blocked`, `crawl_error`, `parse_error` or `load_sitemap`. Use this field for filtering, automation or retry logic |
| `error` | Human-readable message | The detailed failure reason, for example `Crawler error: Cannot find product title` |
| `status_code` | HTTP-like number | A summary of the crawl result. Optional, see the status code table below |
| `warning` | Human-readable message | A non-fatal issue on a record that was still delivered |
| `warning_code` | Machine-readable identifier | The machine-readable version of `warning`, for example `dead_page` or `validation` |
Branch on `error_code` rather than on the raw `error` string, because the message text changes as scrapers are updated:
```js theme={null}
if (line.error_code === 'dead_page') {
// Handle expired or removed URL
}
```
### What do the Scraper Studio status codes mean?
`status_code` summarizes the crawl result as an HTTP-like numeric value.
| Status code | Meaning |
| ----------- | -------------------------------------------------------------------------------------- |
| 400 | Bad request or invalid page response |
| 403 | Access blocked or denied by the target site |
| 404 | Page not found, removed or invalid URL |
| 407 | Account is suspended or access is restricted |
| 408 | Request timed out. The target site did not respond in time |
| 421 | Request was routed to the wrong server |
| 429 | Too many requests. Rate limit reached |
| 500 | Crawl, proxy or navigation failed before Scraper Studio received a valid page response |
| 503 | Target site or service is temporarily unavailable, overloaded or timed out |
`status_code` is sometimes missing or `undefined`, because some error paths do not assign a numeric status code. Treat `status_code` as optional and do not assume it is always present.
## How do errors and warnings differ?
An error means the record failed. A warning means the record was delivered but something should be reviewed.
* **Error:** the record failed and should be treated as unsuccessful. A failed record usually includes `error`, `error_code` and `status_code`
* **Warning:** the record was delivered, but something should be reviewed. A delivered record with a non-fatal issue usually includes `warning`, `warning_code` and `status_code`
* **Successful record:** none of `error`, `error_code`, `warning` or `warning_code` is populated
Warnings occur when a partial crawl returned data but also hit an issue, when a normally failing condition was configured as a non-error, or when output validation found an issue but the record was still kept.
When a `dead_page` error is downgraded to a warning, the output includes:
```text theme={null}
warning_code = dead_page
warning = Dead page detected
```
Schema validation warnings usually use:
```text theme={null}
warning_code = validation
```
The same underlying issue can appear either as an error or as a warning, depending on the scraper logic, validation rules and output schema settings.
## Where do Scraper Studio errors come from?
Scraper Studio errors come from one of three layers.
### What causes scraper errors?
Scraper errors come from the scraper's interaction code, parser code or input handling. Examples include invalid input, `wait_element_timeout`, `parse_error`, `click_timeout`, `dead_page`, `bad_input` and `blocked`.
Fix these by updating the scraper logic, parser selectors, validation rules or input data.
### What causes proxy and unblocker errors?
Proxy and unblocker errors come from Bright Data's proxy, routing or unblocking layer. Examples include proxy connection issues, target website blocking, geo or zone configuration issues, no available peers and rate limiting.
Fix these by retrying, reducing the request rate, changing country or geo settings, or contacting Bright Data support if the issue persists.
### What causes platform and infrastructure errors?
Platform and infrastructure errors come from the Scraper Studio platform, browser worker, parser sandbox, storage layer or internal infrastructure. Examples include worker timeout, browser disconnected, parser memory limit exceeded, upload failure and WebSocket connection issues.
These are often transient and are frequently resolved by retrying. If the issue persists, open a support ticket with the scraper ID, job ID, failed input and raw error message.
## Common scraper error codes
These error codes indicate scraper execution issues. The request may have failed because of invalid input, a missing or changed page element, target blocking, parsing problems, rate limits, CAPTCHA handling or scraper logic that needs to be updated.
| Error code | Meaning | Suggested action |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dead_page` | The scraper detected that the page does not exist or is no longer relevant. | Check whether the page still opens manually. Remove expired URLs or fix discovery logic that produces stale links. |
| `bad_input` | The scraper marked the input as invalid and skipped retries. This happens when required input fields are missing, the URL is invalid or the page is not the expected type. | Validate inputs before triggering the scraper. Remove invalid URLs or update discovery logic to avoid unsupported page types. |
| `blocked` | The `blocked()` function was used in the scraper code, or the target website refused access to the page. | Retry later, reduce the request rate, or review country, session, CAPTCHA and block-handling logic. |
| `crawl_error` | A general crawl execution error occurred. This happens when the page did not load correctly, required page data was missing, the browser session closed or scraper code tried to read missing data. | Open Debug crawl or Crawl inspector for the failed input. Check page loading, add null checks and rerun failed pages. |
| `wait_element_timeout` | The scraper waited for an element that did not appear in time. | Confirm the selector still exists and is stable. Increase the timeout or handle alternate page layouts if the page loads slowly. |
| `ajax_request_error` | A background request failed or timed out during page execution. | Review network requests in preview or debug mode. If the request is optional, relax validation or add fallback logic. |
| `collector_request_validation` | A custom request validation rule rejected an AJAX or network response, usually from `verify_requests()` logic. | Review the validation callback and confirm the rejected response is required. Relax the rule if the request is optional. |
| `captcha_timeout` | CAPTCHA solving did not complete within the expected time. | Retry the crawl and check whether the target repeatedly shows CAPTCHA. Review CAPTCHA handling and job deadline settings. |
| `close_popup_fail` | Scraper Studio failed to close a popup, such as a consent manager or modal. | Check whether the popup selector changed or appears conditionally. Update popup handling or make the scraper tolerate the popup if it does not block extraction. |
| `click_timeout` | A click action did not complete within the timeout. The element may be missing, hidden, disabled or not clickable. | Confirm the selector matches a visible, clickable element. Add `wait(selector)` before clicking or update the click logic. |
| `tag_response` | A tagged response callback failed, or the tagged response returned an unexpected status or format. | Review the tagged response callback and confirm the expected response exists. Check the browser network tab in the IDE to verify that the response loads correctly, and add checks before reading fields from it. |
| `load_sitemap` | Scraper Studio failed to load or parse a sitemap. Causes include fetch failure, request timeout or invalid sitemap format. | Open the sitemap URL manually and confirm it is reachable and valid. |
| `load_more_timeout` | The scraper waited for more items to load, but no new items appeared before the timeout. | Confirm the "load more" selector and list container still exist. Add a stop condition when no more results are available. |
| `child_input_size_validation` | A child input created by `next_stage()` exceeded the allowed input size. | Reduce the data passed to `next_stage()`. Pass only fields needed by the next stage, such as URL, ID or small metadata values. |
| `detect_block` | Scraper Studio detected block-page content, such as "Access Denied", sign-in walls or other known blocking patterns. | Open the failed page manually and confirm whether it shows a block page. If many inputs fail, reduce the request rate or review location and session behavior. |
| `ERR_INVALID_URL` | A URL value is invalid or not in a supported format. | Validate input URLs before triggering the scraper. Make sure URLs include `http://` or `https://` and do not contain malformed characters. |
| `not_supported_cmd` | The scraper used a function that is not supported by the selected worker type. | Change the worker type or replace the function. Use a Browser worker for browser-only actions such as click, scroll, type and wait. See [Worker types](/datasets/scraper-studio/worker-types). |
| `detached_element` | The scraper referenced an element that was removed or re-rendered before the action completed. | Re-select the element immediately before interacting with it. Add waits after page updates and avoid storing element handles across DOM changes. |
| `timeout` | The scraper timed out while waiting for an element, request, response or page action to complete. | Confirm that the expected element or response exists. Increase the timeout only when needed and add fallback logic for optional or slow-loading content. |
### How do `block` and `blocked` differ?
`block` and `blocked` are different error codes. `blocked` is raised by scraper logic, while `block` is raised by the layer that fetched the page.
`blocked` is usually raised by scraper logic. The scraper calls `blocked()` when it detects a blocked page, login wall, CAPTCHA page or other condition that should mark the crawl as blocked:
```js theme={null}
blocked('Login page was shown');
```
`block` usually comes from navigation, request rejection, target-site blocking or proxy-level response handling. It may include the real HTTP status returned by the target site or upstream layer. Observed status codes for `block` are:
```text theme={null}
400, 401, 403, 404, 405, 409, 410, 418, 429, 500, 503
```
In short:
* `blocked` is collector-level block detection, usually raised by scraper logic
* `block` is target, proxy or navigation-level rejection, often with an HTTP status
## Navigation and browser error codes
These error codes indicate navigation or browser loading issues. The scraper may have failed because the page did not load in time, the browser could not reach the URL, the target kept network requests open, or the connection timed out or closed before completion.
| Error code | Meaning | Suggested action |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bad_navigate` | The browser failed to navigate to the requested URL. | Open the URL manually and check whether it redirects or blocks access. Retry the crawl if the failure is temporary. |
| `navigation_timeout` | The page did not finish navigation within the allowed time. | Increase the navigation timeout only if needed. Prefer `wait(selector)` or a different `wait_until` strategy instead of waiting for full page load. |
| `domcontentloaded_event_timeout` | The browser timed out while waiting for the `DOMContentLoaded` event. The page did not reach the expected load state in time. | If full page load is not required, use `navigate(input.url, { wait_until: 'navigate' })` and then `wait(selector)` for the element you need. |
| `networkidle_event_timeout` | The browser timed out while waiting for network activity to become idle. Some pages never reach network idle because they keep background requests open. | Avoid relying on full network idle for pages with continuous background traffic. Use `wait_until: 'navigate'` and wait for a specific selector instead. |
| `load_event_timeout` | The page did not fire the full `load` event in time. | Avoid waiting for full page load if it is not required. Use selector-based waits for the content needed by the parser. |
| `document_load_failed` | The browser started navigation, but the main document failed to load. Causes include an HTTP failure, SSL/TLS issue, empty response, missing credentials, redirect issue or proxy and peer problem. | Retry and verify the target URL opens in a browser. If the failure persists, try a different geo or open a support ticket with the job ID and full error. |
| `net_err_timed_out` | The network request timed out. | Retry and check target stability. Reduce concurrency if the timeout affects many inputs. |
| `net_err_closed` | The target site closed the connection unexpectedly. | Check whether the site is available and retry. If it happens frequently, review blocking or rate behavior. |
| `net_err_cert_date_invalid` | The SSL/TLS certificate presented by the server is not valid for the current date. | Verify that the target URL opens correctly in a browser and that the site certificate is valid. |
| `net_err_http2_protocol_error` | The browser encountered an HTTP/2 protocol error while loading the target page or resource. | Retry the request. If it persists, confirm whether the target site loads normally in a browser. |
| `net_err_cert_authority_invalid` | The target site's SSL/TLS certificate is signed by an untrusted or unknown certificate authority. | Verify that the target URL opens correctly in a browser and that the certificate is trusted. |
### Browser and infrastructure lifecycle errors
These error codes indicate that the browser session, runner or browser-control connection was interrupted during the crawl. They are usually transient platform or browser lifecycle errors.
Common lifecycle error codes are `runner_disconnected`, `network_error`, `cdp_conn_err`, `cdp_cmd_timeout`, `cdp_disconnect`, `bad_browser`, `browser_disconnected` and `ipc_timeout`.
**Suggested action:** retry the job. If the error persists, open a support ticket with the job ID, response IDs, failed input and raw error message.
### Rate limit variants
`global_rate_limit` and `bucket_rate_limit` mean that requests to the target domain are being rate limited. This usually happens when the target site is sensitive to high request volume or too many requests are sent in parallel.
**Suggested action:** retry after a short delay. If the issue repeats, reduce concurrency and queue jobs instead of running many jobs in parallel.
## Job lifecycle error codes
These error codes indicate that the collection did not complete because of runtime, deadline, cancellation or page-volume limits.
| Error code | Meaning | Suggested action |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `job_run_timeout` | The collection or crawl exceeded the allowed runtime. This happens during long waits, slow page actions or CAPTCHA solving. | Optimize scraper logic and check for loops, long waits or repeated CAPTCHA solving. Reduce work per request or increase the deadline if available. |
| `crawl_timeout` | The crawler did not return a result within the internal timeout. | Retry the crawl. Simplify the crawl logic or reduce page complexity if it repeats. |
| `deadline_timeout` | The job reached its configured deadline. | Increase the deadline or reduce work per request. For large workloads, split the job into smaller runs. |
| `too_many_pages` | The job created too many child pages. | Reduce the number of child pages generated per input. Use batch collection for high fan-out jobs. |
| `uncrawled_page` | The page result was requested before the page was crawled, usually because the job deadline was reached. | Increase the deadline or reduce the number of pages created by the scraper. |
| `aborted_page` | The crawl was aborted because the job was canceled. | No action is required if the job was intentionally canceled. Otherwise, rerun the affected inputs. |
## Infrastructure and storage error codes
These error codes indicate platform, worker, storage or delivery issues. The collection may have failed because an internal service was unavailable, a worker was overloaded, the result was too large or delivery to an external destination failed.
| Error code | Meaning | Suggested action |
| ---------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `infra_error` | An internal infrastructure error occurred. | Retry the request. If the issue persists, contact support with the job or response ID. |
| `page_too_big` | The loaded page exceeded the Scraper Studio page size limit of 16 MB. | Collect less data per page. Avoid storing large HTML blobs or unnecessary fields. |
| `crawl_request_failed` | The crawl runner could not connect to a crawler worker after retries. | Retry the job. Escalate to support if the issue persists. |
| `worker_too_busy` | The crawler worker was overloaded. | Retry later and reduce concurrency. |
| `external_upload_fail` | Uploading a result or media file to an external destination failed. | Verify delivery destination configuration, credentials, permissions and path settings. |
| `failed_media_upload` | Scraper Studio failed to save a media file. | Retry the failed page and confirm the media URL is reachable. If the failure persists, check delivery and storage settings or contact support. |
## Proxy and unblocker error codes
These error codes indicate a routing or connection issue between Scraper Studio and the underlying proxy network. They usually happen when the proxy network cannot establish or maintain a stable connection to the target site.
| Error code | Meaning | Suggested action |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `proxy` | A proxy-layer failure occurred during navigation or request. Possible causes include no peer availability, DNS failure, SSL issues, target rejection, timeout or protection pages. | Retry, switch country if needed and verify the target is accessible. If the failure persists, contact support with the job ID and raw error. |
| `proxy_error` | The proxy layer could not connect to the target website or received an unexpected upstream response. | Retry and verify the target URL. If the issue repeats, try another country or escalate with the job ID and error details. |
| `net_err_tunnel` | The browser could not establish a tunnel connection through the proxy. | Check whether the website is available and retry. Escalate if the issue persists across many inputs. |
| `no_peers` | Scraper Studio failed to establish a connection with a peer. | Retry later or remove restrictive peer settings such as country, location or session constraints. |
## Parser and payload error codes
These error codes indicate parser execution or payload-size issues. The scraper may have failed because parser code encountered invalid or missing data, the payload sent to the parser was too large, or parser execution exceeded memory or CPU limits.
| Error code | Meaning | Suggested action |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `parse_error` | Parser code failed while extracting or processing data. Common causes include syntax errors, blocked or invalid content, missing fields, invalid URLs, or reading properties from `null` or `undefined`. | Run a preview with the failing input and inspect the parser code. Add defensive checks and validate selectors. |
| `parse_request_payload_large` | The payload sent to the parser exceeded the size limit. | Reduce the amount of data sent to the parser. Extract only the required HTML section or response fields. |
| `parse_mem_limit_exceeded` | Parser execution exceeded the memory limit, often because of very large HTML, JSON, arrays or embedded data. | Reduce the parser payload and avoid storing large objects in memory. Extract only the required fields. |
| `parse_cpu_limit_exceeded` | Parser execution exceeded the CPU limit, often because of heavy loops, regex or transformations. | Simplify parser logic and reduce loops over large datasets. Move unnecessary processing out of the parser. |
| `parse_req_error` | The parser request payload exceeded the allowed field size. | Reduce the size of the request or parsed payload. Avoid sending very large fields, files or HTML blocks into parser processing. |
## Access and permissions error codes
These error codes indicate an access or compliance issue. The request was stopped because the target is restricted by Bright Data's compliance safeguards or because account permissions need review.
| Error code | Meaning | Suggested action |
| ---------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `brul` | The target is restricted by Bright Data compliance. | If access is required, contact `compliance@brightdata.com` and request permission for the blocked resource. |
## How to troubleshoot a failed Scraper Studio collection
Work through these steps in order when a collection run returns errors.
1. Check the `error_code`.
2. Check the `status_code`, if present.
3. Read the raw `error` message.
4. Open the failed URL manually to confirm whether the page exists and is accessible.
5. Use Debug crawl or Crawl inspector to inspect the failed input, crawl stage, children, files, warnings and output records.
6. If the error is parser-related, run a preview and inspect HTML, Output and Last errors.
7. If many inputs fail with the same error, check the request rate, concurrency, blocking, worker type and recent target-site changes.
8. If the issue persists, open a support ticket with the scraper ID, job or response ID, failed input and raw error message.
## FAQs
### Why is `status_code` missing from my failed record?
Some Scraper Studio error paths do not assign a numeric status code, so `status_code` is sometimes missing or `undefined`. Treat the field as optional and branch on `error_code` instead.
### Does a warning mean the record was lost?
No. A warning means the record was delivered but something should be reviewed, such as a partial crawl, a condition configured as a non-error or an output validation issue. Only records with `error` and `error_code` are unsuccessful.
### Which errors should I retry automatically?
Retry transient platform, browser lifecycle, proxy and rate limit errors, such as `infra_error`, `runner_disconnected`, `browser_disconnected`, `worker_too_busy`, `proxy_error`, `global_rate_limit` and `bucket_rate_limit`. Do not retry `bad_input`, `ERR_INVALID_URL` or `dead_page`, because those need an input or discovery-logic fix.
# Scraper Studio FAQs
Source: https://docs.brightdata.com/datasets/scraper-studio/faqs
FAQs about Bright Data Scraper Studio: 2 scraper types, inputs and outputs, billing, limits, snapshots and reporting issues for custom scraper builds.
This page answers the questions the Bright Data support team hears most often about Scraper Studio. If you need a walkthrough instead of a quick answer, start with [Understanding Scraper Studio](/datasets/scraper-studio/introduction).
## Frequently asked general questions
A Bright Data web scraper is an automated script that collects public web data at scale through Bright Data's proxy and unblocking infrastructure. It returns the collected data in a structured format (JSON, NDJSON, CSV, XLSX) and can deliver it to an API endpoint, webhook, cloud storage, or SFTP. Bright Data maintains hundreds of pre-built scrapers for popular sites in the [Scrapers Library](https://brightdata.com/products/web-scraper).
Bright Data Scraper Studio is a cloud-hosted environment for building custom scrapers. It offers two modes: an AI Agent that generates a scraper from a natural-language description, and an IDE where you write JavaScript directly. Both modes run on the same Bright Data proxy and unblocking infrastructure. See [Understanding Scraper Studio](/datasets/scraper-studio/introduction).
The [Scrapers Library](https://brightdata.com/products/web-scraper) contains pre-built scrapers Bright Data maintains for popular sites such as Amazon, LinkedIn, and Instagram. Bright Data Scraper Studio is the environment you use to build custom scrapers when the site you need is not in the library.
Yes. A single scraper can navigate to any URL you pass in as input. If you need different extraction logic per site, use multiple stages (`next_stage()`) or build one scraper per site.
You do not have to provide every URL. The Bright Data Scraper Studio AI Agent builds four scraper types, three of which find items for you:
* **PDP**: give a list of product URLs you already have; get one row per URL.
* **Discovery**: give a category or listing URL; get one row per item on that listing.
* **Discovery + PDP**: give a category or listing URL; get full per-product detail for every item.
* **Search**: give a keyword and optional country when you have no URLs at all.
Each scraper is scoped to one data shape, so the AI Agent is not a general crawler; do not pass a homepage and ask for "everything." For deeper link discovery across a site, build a multi-stage scraper in the Bright Data Scraper Studio IDE with `next_stage()`, `rerun_stage()` and `load_sitemap()`. See [What can the AI Agent build?](/datasets/scraper-studio/ai-agent#what-can-the-ai-agent-build) and [Functions](/datasets/scraper-studio/functions).
Bright Data Scraper Studio scrapes the web, not native mobile apps. For mobile layouts, the Browser worker can emulate a mobile device with `emulate_device('iPhone X')` so the target site serves its mobile web pages. Native iOS or Android app data is out of scope. See [Functions](/datasets/scraper-studio/functions).
## Inputs, outputs, and schemas
An input is the parameter set Bright Data Scraper Studio passes into the scraper for a single run. Typical inputs are a URL, a search keyword, a product ID or ASIN, a profile handle, or a date range. Multiple inputs can be passed in one job via CSV upload or the API.
The output is the structured data the scraper returns for an input. Bright Data Scraper Studio delivers output in JSON, NDJSON, CSV, XLSX, or Parquet based on the scraper's delivery preferences.
One input can generate multiple records. For example, if you submit 5 product listing URLs and each listing page contains 20 products, the scraper returns 100 records from 5 inputs. The statistics page counts records, not inputs.
A search scraper takes a keyword as input instead of a URL. Bright Data Scraper Studio runs a search on the target site and extracts data from the result pages. Use a search scraper when you do not have specific URLs to scrape.
A discovery scraper collects data from listing pages such as search results, category pages, or directories. It extracts fields that appear directly on the listing (titles, prices, ratings) and can also collect product URLs or IDs for a follow-up product-page scrape.
When the input or output schema changes, the scraper must be updated to match. If you trigger the scraper before Bright Data has updated it, you will see an `input(output)_schema_incompatible` error.
To trigger the scraper anyway and ignore the schema mismatch, click **Trigger anyway** in the UI or add a parameter to your API request:
* Output schema incompatible: `override_incompatible_schema=1`
* Input schema incompatible: `override_incompatible_input_schema=1`
```bash theme={null}
curl "https://api.brightdata.com/dca/trigger?scraper=ID_COLLECTOR&queue_next=1&override_incompatible_schema=1" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer API_KEY" \
-d '[{"url":"https://targetwebsite.com/product_id/"}]'
```
## Building and editing scrapers
You have three options in Bright Data Scraper Studio:
* **Build with the AI Agent**: describe the data you want in plain language. See [Scraper Studio AI Agent](/datasets/scraper-studio/ai-agent).
* **Build in the IDE**: write JavaScript directly. See [Develop a scraper with the IDE](/datasets/scraper-studio/develop-a-scraper).
* **Request a managed scraper**: Bright Data's data team builds and maintains the scraper for you.
Yes. Every scraper, including ones the AI Agent generated, can be opened and edited in the Bright Data Scraper Studio IDE. You can change extraction logic, modify selectors, add or remove output fields, and tune performance. If you prefer not to write code, use the [Self-Healing tool](/datasets/scraper-studio/self-healing-tool) to request changes in plain language.
Pass the AI Agent a target URL and an optional description of the data you want. The agent asks clarifying questions, generates an output schema for your review, and writes the full scraper code once you approve the schema. You can then run the scraper immediately or schedule recurring runs. See [Scraper Studio AI Agent](/datasets/scraper-studio/ai-agent) for a full walkthrough.
Yes. You own every line of the scraper, including code the AI Agent generates, and you can read and edit it in the Bright Data Scraper Studio IDE. Scrapers are plain JavaScript, so you can copy the code out and keep it in your own repository. To run a scraper outside Bright Data you would need to supply your own proxy and unblocking infrastructure.
Yes. Choose the Browser worker for sites that render content with JavaScript; it runs a full headless browser so you can wait for elements, click, scroll and capture background network calls. Choose the Code worker for static HTML and HTTP responses, which is faster and cheaper. See [Worker types](/datasets/scraper-studio/worker-types).
## Self-Healing
You trigger Self-Healing; it does not fire automatically when a target site's DOM changes. You can trigger it three ways:
* **Control panel**: open the scraper in the Bright Data Scraper Studio IDE, describe the fix in plain language and accept the AI-generated diff. See [Self-Healing tool](/datasets/scraper-studio/self-healing-tool).
* **Bright Data CLI**: run `bdata scraper heal ` then `bdata scraper approve`. Add `--auto-approve` for unattended workflows. See [Build a scraper with the Bright Data CLI](/datasets/scraper-studio/build-with-the-cli#how-do-i-fix-a-scraper-when-the-site-changes).
* **Bright Data API**: call `POST /dca/collectors/{collector_id}/refactor_template`. See [Trigger Self-Healing](/api-reference/scraper-studio-api/ai-flow/trigger-self-healing).
Each path repairs the code you own and keeps the same Collector ID. The refactor runs on Bright Data infrastructure, not on your machines, and can take up to 15 minutes; Bright Data emails you when the diff is ready.
## Running and triggering scrapers
Bright Data Scraper Studio supports three trigger methods:
* **By API**: regular request, queue request, or replace request
* **Manually**: from the control panel
* **On a schedule**: hourly, daily, weekly, or custom
See [Initiate collection and delivery](/datasets/scraper-studio/initiate-collection-and-delivery-options).
A queued request tells Bright Data Scraper Studio to wait until the previous request for the same scraper finishes before starting the next one. Use it when you want serial execution instead of running multiple jobs in parallel.
Bright Data Scraper Studio runs up to **100 batch jobs in parallel** per scraper. Additional jobs queue automatically and start as capacity frees up. See [Scraper Studio specifications](/datasets/scraper-studio/specifications) for full limits.
In the Bright Data Scraper Studio dashboard, click the **Bug** icon under **Failed crawls** to open the scraper in the IDE. Failed inputs appear in the **Last errors** tab with the exact error message and error code. Bright Data stores the last 1,000 errors per virtual job so you can re-run failed inputs and diagnose the issue.
## Blocking and unblocking
Bright Data Scraper Studio runs every scraper on Bright Data's proxy and unblocking network, with automatic retries and real-user fingerprinting built in. The IDE provides `solve_captcha()` to solve CAPTCHAs, `detect_block()` and `blocked()` to detect and report blocks, and `country()` to route a request through a specific country when a site is geo-restricted. See [Functions](/datasets/scraper-studio/functions).
## Billing and limits
CPM stands for "cost per mille", meaning 1,000 page loads. Bright Data Scraper Studio bills page loads in CPM units. Current rates are on the [pricing page](https://brightdata.com/pricing/web-scraper/custom).
A billable event is any function that causes Bright Data Scraper Studio to load a page or perform a network request:
* `navigate()`
* `request()`
* `load_more()`
* Media file download (billed per GB, separate from CPM)
Scraper Studio does not have a separate page-load or record allotment, but it is covered by the account-level free tier. Every new account gets 5,000 free credits per month, a single shared pool usable across the Web Unlocker API, SERP API, Web Scraper API and Scraper Studio. In the free tier, Scraper Studio consumes one credit per page load drawn from that pool; records are not billed by default. New accounts also get a one-time onboarding bonus credit usable on any Bright Data product. See the [free tier](/general/account/billing-and-pricing/free-tier) and [billing](/general/account/billing-and-pricing/billing) pages.
## Snapshots and data retention
Snapshot retention depends on the collection type:
* **Batch collections**: 16 days
* **Real-time collections**: 7 days
After that, snapshots are permanently deleted. Bright Data does not recover expired data. Download or export your data before the retention window closes, or configure the scraper to deliver results automatically via webhook, API download, or cloud storage.
## How to report issues
Open the scraper in the Bright Data Scraper Studio control panel and select **Report an issue** from the three-dots menu. Bright Data routes the ticket to a different team based on the issue type:
* **Data** (missing fields, missing records, parsing errors): routed to the scraper engineer. Available only for managed scrapers.
* **Collection and delivery** (incomplete delivery, slow scraper): routed to the support team.
* **Other** (UI issue, product question): routed to the account manager.
Include the affected job ID, a description of the problem, and a screenshot or file if it helps show the issue.
Include:
* The issue category (wrong data, missing records, delivery problem, IDE issue, other)
* A description of the exact problem
* The affected job ID
* A screenshot or file that shows the issue, if possible
Bright Data opens a ticket automatically and the R\&D team handles it.
You receive an email when a Bright Data engineer starts building the scraper and another email when it is ready. You can also track the status on the Scrapers dashboard.
## Related
How Bright Data Scraper Studio works and when to use it
Infrastructure limits, billing, and data retention
# Scraper Studio dashboard features
Source: https://docs.brightdata.com/datasets/scraper-studio/features
Reference for the Bright Data Scraper Studio dashboard: scraper types, statuses, run statistics columns and how to download collected data.
The Bright Data Scraper Studio dashboard is where every scraper lives. Use it to inspect scraper properties, trigger runs, configure delivery, manage versions and review collection statistics. This page is a reference for every control you see on the **My Scrapers** page and the statistics drill-down.
## The Scrapers dashboard
Every scraper you use, whether generated by the AI Agent, built in the Bright Data Scraper Studio IDE, or built by Bright Data teams, appears under **My Scrapers** on the Scrapers dashboard.
The **My Scrapers** page lists every scraper type, not only Scraper Studio scrapers.
### Which scraper types appear on the dashboard?
The **My Scrapers** page lists three scraper types, distinguished by who builds and maintains each scraper.
| Type | Description |
| -------------------------- | ------------------------------------------------------------ |
| **Scraper API** | 700+ pre-built scrapers, built and maintained by Bright Data |
| **Scraper Studio** | Scrapers built and maintained by customers or AI |
| **Scraper Studio Managed** | Scrapers built and maintained by the Bright Data team |
### What does each scraper status mean?
Each scraper and run carries a status that shows where it is in the build and collection lifecycle.
| Status | Description |
| ------------ | ---------------------------------------------------------------------------------------- |
| **Ready** | The run has been completed |
| **Canceled** | The run was canceled by the customer or system |
| **Active** | The scraper is saved to production and ready to initiate, but has not been triggered yet |
| **Draft** | The scraper is still being built and is not yet saved to production |
| **Running** | The collection is running |
To open a specific scraper's dashboard, click the scraper.
## What does the collection statistics page show?
The statistics page shows per-collection metrics for every run of a scraper.
| Column | Description |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **ID** | Unique identifier for the collection job |
| **Trigger** | Who triggered the job and how (API, UI, or schedule) |
| **Template** | Template version number |
| **Inputs** | Number of inputs submitted to the collection |
| **Records** | Number of records the scraper returned |
| **Failed crawls** | Number of pages that failed to crawl |
| **Fulfillment** | Total number of unique pages discovered during scraping, including pages that were found but not necessarily scraped |
| **Success rate** | Percentage of inputs that produced successful records |
| **Queued at** | Timestamp when the job entered the queue |
| **Started at** | Timestamp when the scraper started collecting |
| **Finished at** | Timestamp when the scraper finished collecting |
| **Job time** | Total job duration |
| **Estimated time left** | Time remaining for an in-progress job |
| **Queue** | Name of the queue the job is running in (from the trigger's queue name) |
| **Page Loads** | Total page loads consumed by the job |
| **Spent** | Total amount spent on this collection run |
## Collection run actions
The three-dot **Actions** menu includes the following options. Available actions may vary depending on the run status and delivery configuration.
### **Redeliver results**
Use **Redeliver results** to send collected results again to the configured delivery destination.
You can choose:
* **Dataset only**, redelivers only the structured output data.
* **Dataset + media**, redelivers the structured output data together with collected media files, such as images, videos, PDFs, or documents.
Use this when the original delivery failed, the destination was unavailable, or you updated the destination and want to send the same results again.
### **Rerun or cancel a collection**
Use **Rerun job** to start a new run from an existing collection.
You can choose:
* **Rerun all from beginning**, starts a new run using all original input values.
* **Rerun failed pages**, starts a new run only for pages that failed in the selected collection.
Use **Cancel collection** to stop an active collection before it finishes. Data already collected may still be available, depending on the run status and delivery settings.
### **Inspect results and errors**
Use **Quick view** to preview up to 100 collection records in JSON format directly in the control panel.
Use **Errors** to review failed inputs and error details.
Use **Activity** to monitor a collection run over time. The page shows run status, start and finish times, success rate, crawl count, and performance charts. You can change the visible time range, such as **5 min**, **1 h**, **1 d**, or **all**.
The Activity page includes:
* **Run rate** - shows how many pages were crawled, failed, or retried at each point in time, along with the success rate.
* **Pages** - shows discovered pages, open pages, retriable pages, and pages currently being crawled.
* **Inputs** - shows how many input rows were submitted and prepared for crawling.
* **Crawl rate** - shows the system-calculated maximum number of allowed workers and the actual number of workers assigned to the run.
* **Crawl time per page** - shows the average time spent crawling each page.
Use **Activity** to investigate slow runs, error spikes, retries, or changes in crawl performance.
### **Debug crawl**
Use **Debug crawl** to investigate failed or unexpected scraper behavior.
## How do I download the data from the dashboard?
Click the **Download file options** button, select a file format, and the download starts immediately.
Real-time job outputs are stored on Bright Data's side but cannot be downloaded from the dashboard. To retrieve them, use the Scraper Studio API or send them to one of the delivery destinations configured in your delivery options.
## How do I trigger a scraper and get results?
To trigger a scraper, use the three initiation options documented on [Initiate collection and delivery](/datasets/scraper-studio/initiate-collection-and-delivery-options):
* [Initiate by API](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
* [Initiate manually](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
* [Schedule a scraper](/datasets/scraper-studio/initiate-collection-and-delivery-options#how-do-i-trigger-a-scraper-run)
To retrieve results, click the three-dot icon on a scraper row and select **Statistics** to open the collection statistics page, where you can download the data.
## Related
Configure trigger methods, file formats and delivery destinations
Reference for every panel and control in the IDE
# Scraper Studio functions reference
Source: https://docs.brightdata.com/datasets/scraper-studio/functions
Reference for Bright Data Scraper Studio interaction and parser functions: navigation, waits, selectors and parser helpers with parameters and 30+ examples.
This reference documents every function available in Bright Data Scraper Studio's IDE: the interaction code that controls a browser session, and the parser code that turns HTML into structured records. Each function lists its parameters, return value, and a runnable example.
Functions marked with **⭐** work only with the Browser worker and throw an error when called from a Code worker. See [Browser-only functions](#browser-only-functions) for the full list.
## How is Scraper Studio code organized?
A Bright Data Scraper Studio scraper uses two code types:
| Code type | Role | Language and libraries |
| ---------------- | ----------------------------------------------------------------------------------------------- | ----------------------------------------- |
| Interaction code | Navigates the target site: URL requests, clicks, scrolls, waits, and background traffic capture | JavaScript + Bright Data browser commands |
| Parser code | Extracts and structures data from the HTML returned by interaction code | JavaScript + Cheerio (`$`) |
You move data from one to the other with `parse()` (which runs the parser) and `collect()` (which appends a record to the final dataset).
## Interaction functions
Interaction functions run in the scraper's main JavaScript context and drive the browser or HTTP client. Use them to navigate, wait for elements, interact with the page, capture network traffic, and hand off data to the parser.
### Global objects
| Name | Type | Description |
| ---------- | ------ | -------------------------------------------------------------------------------------------------------------------------- |
| `input` | object | Input for the current stage, set by the trigger or by a previous `next_stage()`/`run_stage()`/`rerun_stage()` call. |
| `job` | object | Metadata about the current job (for example `job.created`, the job start timestamp). |
| `location` | object | Info about the current browser location. Field: `href`. |
| `parser` | object | Values captured by `tag_response`, `tag_script`, and related tagging functions, available after `wait_for_parser_value()`. |
```js theme={null}
navigate(input.url);
let {created} = job;
console.log('current url', location.href);
```
### Navigation
#### `navigate`, Load a URL in the browser
Navigates the browser to a URL. A 404 status throws a `dead_page` error by default; override with `allow_status`.
**Parameters**
| Parameter | Type | Required | Default | Description |
| ------------------ | ------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------- |
| `url` | string or URL | Yes | , | Target URL |
| `opt.wait_until` | string | No | `load` | `load`, `domcontentloaded`, `networkidle0`, or `networkidle2` |
| `opt.timeout` | number | No | `30000` | Navigation timeout in milliseconds |
| `opt.referer` | string | No | , | `Referer` header to send |
| `opt.allow_status` | number\[] | No | `[]` | HTTP status codes to accept without throwing |
| `opt.fingerprint` | object | No | , | Override browser fingerprint (`screen.width`, `screen.height`) |
| `opt.sniff_mime` | boolean | No | `false` | Detect the MIME type and store it as `sniffed_mime`. On code workers, only `text/html` and `application/pdf` are detected |
```js theme={null}
navigate(input.url);
navigate('https://example.com');
navigate('https://example.com', {wait_until: 'domcontentloaded'});
navigate('https://example.com', {referer: 'https://google.com'});
navigate('https://example.com', {timeout: 45000});
navigate('https://example.com', {allow_status: [404]});
navigate('https://example.com', {
fingerprint: {screen: {width: 400, height: 400}},
});
navigate('https://example.com', {sniff_mime: true});
```
#### `request`, Make a direct HTTP request
Sends an HTTP request without using a browser. Use on Code worker, or on Browser worker when you want to bypass the browser.
**Parameters**
| Parameter | Type | Required | Description |
| ------------------ | ---------------- | -------- | ---------------------------------------------------------------- |
| `url` \| `options` | string or object | Yes | URL string, or an object with `url`, `method`, `headers`, `body` |
```js theme={null}
let res = request('https://www.example.com');
let res = request({
url: 'https://www.example.com',
method: 'POST',
headers: {'Content-type': 'application/json'},
body: {hello: 'world'},
});
```
#### `next_stage`, Queue input for the next stage
Runs the next stage of the scraper in a new browser session with the given input.
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ------------------------------------- |
| `input` | object | Yes | Input object passed to the next stage |
```js theme={null}
next_stage({url: 'https://example.com', page: 1});
```
#### `run_stage`, Run a specific stage
Runs a named stage of the scraper in a new browser session.
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | --------------------------------- |
| `stage` | number | Yes | Stage index (starts at 1) |
| `input` | object | Yes | Input object passed to that stage |
```js theme={null}
run_stage(2, {url: 'https://example.com', page: 1});
```
#### `rerun_stage`, Re-run the current stage with new input
Runs this stage again with a new input. Use it to fan out work (for example, one re-run per page in a pagination).
```js theme={null}
rerun_stage({url: 'https://example.com/other-page'});
```
#### `load_sitemap`, Read URLs from an XML sitemap
Loads a sitemap XML file and returns the URL list. Supports sitemap indexes and gzip-compressed sitemaps.
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | ------ | -------- | ----------- |
| `options.url` | string | Yes | Sitemap URL |
```js theme={null}
let {pages} = load_sitemap({url: 'https://example.com/sitemap.xml.gz'});
let {children} = load_sitemap({url: 'https://example.com/sitemap-index.xml'});
```
#### `resolve_url`, Follow a URL through redirects
Returns the final URL that the given URL argument leads to.
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------------- | -------- | -------------- |
| `url` | string or URL | Yes | URL to resolve |
```js theme={null}
let {href} = parse().anchor_elem_data;
collect({final_url: resolve_url(href)});
```
#### `redirect_history`, Get the redirect chain
Returns the history of URL redirects since the last `navigate()` call.
```js theme={null}
navigate('http://google.com');
let redirects = redirect_history();
// ['http://google.com', 'http://www.google.com', 'https://www.google.com/']
```
#### `response_headers`, Read the last response headers
Returns the response headers from the last page load.
```js theme={null}
let headers = response_headers();
console.log('content-type', headers['content-type']);
```
#### `status_code`, Read the last response status
Returns the HTTP status code of the last page load.
```js theme={null}
collect({status_code: status_code()});
```
### Waiting on the page ⭐
All wait functions are Browser worker only.
#### ⭐ `wait`, Wait for an element to appear
**Parameters**
| Parameter | Type | Required | Default | Description |
| ------------- | ------- | -------- | ------- | ---------------------------------------------------- |
| `selector` | string | Yes | , | CSS selector to wait for |
| `opt.timeout` | number | No | `30000` | Timeout in milliseconds |
| `opt.hidden` | boolean | No | `false` | Wait for the element to be hidden instead of visible |
| `opt.inside` | string | No | , | Selector of an iframe to look inside |
```js theme={null}
wait('#welcome-splash');
wait('.search-results .product');
wait('[href^="/product"]');
wait('#welcome-splash', {timeout: 5000});
wait('#welcome-splash', {hidden: true});
wait('#welcome-splash', {inside: '#iframe_id'});
```
#### ⭐ `wait_any`, Wait for any of several conditions
Waits for any matching condition to succeed. Returns when the first selector resolves.
```js theme={null}
wait_any(['#title', '#notfound']);
```
#### ⭐ `wait_visible`, Wait for an element to be visible
**Parameters**
| Parameter | Type | Required | Default | Description |
| ------------- | ------ | -------- | ------- | ----------------------- |
| `selector` | string | Yes | , | CSS selector |
| `opt.timeout` | number | No | `30000` | Timeout in milliseconds |
```js theme={null}
wait_visible('#welcome-splash');
wait_visible('#welcome-splash', {timeout: 5000});
```
#### ⭐ `wait_hidden`, Wait for an element to disappear
**Parameters**
| Parameter | Type | Required | Default | Description |
| ------------- | ------ | -------- | ------- | ----------------------- |
| `selector` | string | Yes | , | CSS selector |
| `opt.timeout` | number | No | `30000` | Timeout in milliseconds |
```js theme={null}
wait_hidden('#welcome-splash');
wait_hidden('#welcome-splash', {timeout: 5000});
```
#### ⭐ `wait_for_text`, Wait for text content
Waits for an element on the page to contain the given text.
**Parameters**
| Parameter | Type | Required | Description |
| ---------- | ------ | -------- | ---------------- |
| `selector` | string | Yes | CSS selector |
| `text` | string | Yes | Text to wait for |
```js theme={null}
wait_for_text('.location', 'New York');
```
#### `wait_for_parser_value`, Wait for a parser field to be populated
Use after `tag_response()` or `tag_script()` to wait until the captured data is available.
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | -------- | -------- | ---------------------------------------------------------- |
| `field` | string | Yes | Parser field path to wait on |
| `validate_fn` | function | No | Optional callback returning `true` when the value is valid |
| `opt.timeout` | number | No | Timeout in milliseconds |
```js theme={null}
wait_for_parser_value('profile');
wait_for_parser_value('listings.0.price', v => parseInt(v) > 0, {timeout: 5000});
```
#### ⭐ `wait_network_idle`, Wait until the browser network settles
Waits until the browser network has been idle for a given period.
**Parameters**
| Parameter | Type | Required | Default | Description |
| ------------- | ------ | -------- | ------- | --------------------------------------------------- |
| `opt.timeout` | number | No | `500` | Milliseconds of idleness required |
| `opt.ignore` | array | No | `[]` | Patterns (string or RegExp) for requests to exclude |
```js theme={null}
wait_network_idle();
wait_network_idle({
timeout: 1e3,
ignore: [/long_request/, 'https://example.com'],
});
```
#### ⭐ `wait_page_idle`, Wait until DOM mutations stop
Waits until no changes are made to the DOM tree for a given period.
**Parameters**
| Parameter | Type | Required | Description |
| ------------------ | ------ | -------- | --------------------------------------------- |
| `opt.idle_timeout` | number | No | Milliseconds of stability required |
| `opt.ignore` | array | No | Selectors to exclude from mutation monitoring |
```js theme={null}
wait_page_idle();
wait_page_idle({
ignore: ['.live-clock', '.carousel'],
idle_timeout: 1000,
});
```
### Element interaction ⭐
All interaction functions require Browser worker.
#### ⭐ `click`, Click an element
Clicks an element, waiting for it to appear first.
**Parameters**
| Parameter | Type | Required | Description |
| ----------------- | --------------- | -------- | ------------------------------------------------- |
| `selector` | string or array | Yes | CSS selector or Shadow DOM selector path |
| `opt.coordinates` | `{x, y}` | No | Click the closest match to given page coordinates |
```js theme={null}
click('#show-more');
$('#show-more').click();
// Click the map pin closest to the center of a map
let box = bounding_box('#map');
let center = {x: (box.left + box.right) / 2, y: (box.top + box.bottom) / 2};
click('.map-pin', {coordinates: center});
```
#### ⭐ `right_click`, Right-click an element
Same as `click` but uses the right mouse button.
```js theme={null}
right_click('#item');
```
#### ⭐ `hover`, Hover over an element
Moves the cursor over an element, waiting for it to appear first.
```js theme={null}
hover('#item');
```
#### ⭐ `mouse_to`, Move the cursor to a coordinate
**Parameters**
| Parameter | Type | Required | Description |
| --------- | ------ | -------- | ----------------- |
| `x` | number | Yes | Target X position |
| `y` | number | Yes | Target Y position |
```js theme={null}
mouse_to(0, 0);
```
#### ⭐ `type`, Enter text into an input
Waits for the input to appear, then types the given text.
**Parameters**
| Parameter | Type | Required | Description |
| ------------- | --------------- | -------- | ----------------------------------------------------- |
| `selector` | string | Yes | CSS selector |
| `text` | string or array | Yes | Text to type, or an array of strings and special keys |
| `opt.replace` | boolean | No | Clear existing text before typing |
```js theme={null}
type('#location', 'New York');
type('#location', 'New York', {replace: true});
type('[id$=input-box]', 'search term');
type('#search', ['Some text', 'Enter']);
type('#search', ['Backspace']);
```
#### ⭐ `press_key`, Press a special key
Types special keys like Enter or Backspace in the currently focused input.
```js theme={null}
press_key('Enter');
press_key('Backspace');
```
#### ⭐ `select`, Pick a value from a select element
**Parameters**
| Parameter | Type | Required | Description |
| ---------- | ------ | -------- | ------------------------------------ |
| `selector` | string | Yes | CSS selector of a `