Filter dataset

Run async filter jobs on 900+ Bright Data Marketplace datasets. Returns a snapshot_id to download, with CSV/JSON uploads up to 200 MiB.

POST /datasets/filter

cURL

cURL

curl --request POST \
  --url 'https://api.brightdata.com/datasets/filter?dataset_id=gd_l1vikfnt1wgvvqz95w' \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{"filter": {"name": "url", "operator": "=", "value": "https://www.instagram.com/natgeo/"}, "records_limit": 10}'

200

{
  "snapshot_id": "<string>"
}

The Filter endpoint of the Bright Data Marketplace Dataset API runs a large or file-based filter job against any of 900+ Marketplace datasets datasets and returns a snapshot_id you can download once the job completes.

Paste your API key into the authorization field. To get an API key, create an account and learn how to generate an API key.

When should I use Filter?

Use Filter for bulk or file-driven jobs where asynchronous processing is acceptable:

• Filter a Marketplace dataset by field values, such as country, price or a boolean flag.
• Find records whose text fields contain a value.
• Match records against a list of exact values.
• Filter against large value lists from CSV or JSON files, such as excluding 100k+ company IDs.
• Bulk exports of more than 1,000 records.
• Datasets not yet supported by Search.

For sub-second real-time lookups on supported datasets, use Search instead.

How does Filter work?

• A call to the Filter endpoint starts an async job and creates a snapshot of the filtered data in your account.
• The endpoint returns a snapshot_id, not the filtered records.
• The maximum job time is 5 minutes. Jobs that run longer are cancelled.
• Charges apply per record in the snapshot, at the standard Marketplace rate of $2.5 CPM.
• Filter works on all 900+ Marketplace datasets.
• Filter groups support a maximum nesting depth of 3 levels.

How do I authenticate?

Filter uses Bearer token authentication. Pass your API key in the Authorization header:

Authorization: Bearer YOUR_API_KEY

Get your key from account settings.

What are the Filter limits?

Limit	Value	Description
Max rows per file	10,000	Each uploaded CSV/JSON file can contain up to 10,000 data rows. The header row is not counted.
Max files per request	No limit	Attach as many files as needed in one multipart request, as long as the total stays within the 200 MiB cap.
Max request size	200 MiB	Total size of all uploaded files and form data combined. Requests over 200 MiB are rejected.
Job timeout	5 minutes	If filtering does not complete within 5 minutes the job is cancelled.
Filter nesting depth	3 levels	Maximum depth for nested filter groups using and/or.
Max parallel jobs	25 per dataset	Up to 25 Filter jobs can be running (snapshot status building) at once per dataset. New requests return 429 until a running job finishes.
Rate limit	5 requests/minute, 120 requests/hour	Maximum number of Filter API calls. Both limits apply.

For complex filtering needs beyond these limits, contact your account manager.

How do I filter with a JSON request?

Use JSON mode when the filter values fit directly in the request. Send dataset_id, records_limit and filter in the JSON body and set Content-Type to application/json:

curl -X POST "https://api.brightdata.com/datasets/filter" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "gd_l1viktl72bvl7bjuj0",
    "records_limit": 100,
    "filter": {
      "name": "name",
      "operator": "includes",
      "value": "John"
    }
  }'

Filter returns a snapshot_id:

{ "snapshot_id": "s_m123456789" }

Which request fields does Filter accept?

Field	Type	Required	Description
dataset_id	string	Yes	ID of the Marketplace dataset to filter. In multipart mode, send it as a query parameter.
records_limit	integer	No	Maximum number of records to include in the snapshot. In multipart mode, send it as a query parameter.
filter	object	Yes	A single field filter, a filter group or a null check.
filter.name	string	Yes	Dataset field to filter by. Use Get dataset metadata to list field names.
filter.operator	string	Yes	How the field is matched against value, for example =, in, includes or >=.
filter.value	string, number, boolean, array or filename	Yes, except for is_null and is_not_null	The value the operator compares against. In multipart mode, the name of an uploaded file.

How do I filter by field type?

Each field filter combines a field name, an operator and a value. The examples below show common operators by field type. For the full operator table, filter groups and nesting rules, see the filter syntax reference.

Text fields

Use includes to match records where the field contains the value:

{ "name": "name", "operator": "includes", "value": "John" }

This filter matches values such as John Smith, Johnson and Senior John Account Manager. Use not_includes to exclude them.

Lists of exact values

Use in to match records where the field value is exactly one of the values in a list:

{ "name": "country", "operator": "in", "value": ["US", "GB", "DE"] }

Use not_in to exclude records whose field value is in the list.

When should I use includes instead of in?

Operator	Use when	Match behavior
includes	You want to match text that contains a value	Partial match
in	You have a defined list of allowed values	Exact match only

For example, includes with engineer matches Sales Engineer and Engineering Manager, while in with ["Sales Engineer"] matches only Sales Engineer.

Boolean fields

Use = with true or false:

{ "name": "verified", "operator": "=", "value": true }

Number fields

Number fields support =, !=, <, <=, >, >=, in, not_in, is_null and is_not_null:

{ "name": "price", "operator": ">=", "value": 100 }

Date ranges

Combine >= and <= in an and group to match a date range:

{
  "operator": "and",
  "filters": [
    { "name": "created_at", "operator": ">=", "value": "2026-01-01" },
    { "name": "created_at", "operator": "<=", "value": "2026-01-31" }
  ]
}

Array fields

Use array_includes to match records where a multi-value field contains an exact value:

{ "name": "categories", "operator": "array_includes", "value": "Electronics" }

This filter matches records where the categories array contains Electronics. Use not_array_includes to exclude them.

Empty fields

Use is_not_null to keep only records where the field has a value, or is_null for records where the field is empty. These operators take no value:

{ "name": "reviews_count", "operator": "is_not_null" }

How do I filter with CSV or JSON file uploads?

Use file uploads when you need to filter against a large list of values. Send the request as multipart/form-data, pass dataset_id and records_limit as query parameters, and set each filter’s value to the name of an uploaded file:

curl -X POST "https://api.brightdata.com/datasets/filter?dataset_id=gd_l1vijqt9jfj7olije&records_limit=100" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F 'filter={"operator":"and","filters":[{"name":"industries:value","operator":"includes","value":"industries.csv"}]}' \
  -F 'files[]=@/path/to/industries.csv'

The response is the same snapshot_id object that JSON mode returns.

How do I format the CSV or JSON file?

CSV

• The first line must be a header that matches the field name in your filter.
• Each following line contains a single value.
• The file must contain one column only.
• Empty rows and rows that contain only spaces are not supported.

Example: industries.csv

industries:value
Accounting
Ad Network
Advertising

JSON

• The file must be an array of objects, each with a key that matches the field name in your filter.

Example: industries.json

[
  {"industries:value": "Accounting"},
  {"industries:value": "Ad Network"},
  {"industries:value": "Advertising"}
]

Values in the file must match the dataset field format exactly.

Which operators accept a file reference?

File references work only with in, not_in, includes, not_includes, array_includes and not_array_includes. For what each operator does with a file, see the filter syntax reference.

How do I filter against more than 10,000 values?

Split the values into files of up to 10,000 rows each and attach them all in a single request. This example excludes company IDs listed across three files:

curl -X POST "https://api.brightdata.com/datasets/filter?dataset_id=gd_l1vijqt9jfj7olije&records_limit=5000" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F 'filter={"operator":"and","filters":[{"name":"company_id","operator":"not_in","value":"exclude1.csv"},{"name":"company_id","operator":"not_in","value":"exclude2.csv"},{"name":"company_id","operator":"not_in","value":"exclude3.csv"}]}' \
  -F 'files[]=@exclude1.csv' \
  -F 'files[]=@exclude2.csv' \
  -F 'files[]=@exclude3.csv'

Files can also feed different fields in one request. This example combines an industries file and a regions file:

curl -X POST "https://api.brightdata.com/datasets/filter?dataset_id=gd_l1vijqt9jfj7olije" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: multipart/form-data" \
  -F 'filter={"operator":"and","filters":[{"name":"industries:value","operator":"includes","value":"industries.csv"},{"name":"region","operator":"in","value":"regions.csv"}]}' \
  -F 'files[]=@/path/to/industries.csv' \
  -F 'files[]=@/path/to/regions.csv'

How do I fix file upload errors?

Error or issue	Cause	Resolution
Only one column in header is allowed	The CSV contains more than one column.	Use a single-column CSV file.
One or more lines in the input file contain only space characters and cannot be used for filtering. Please remove any empty or whitespace-only line and try again.	The CSV contains empty or whitespace-only rows.	Remove blank rows and rows that contain only spaces.
File exceeds allowed size	The request is larger than 200 MiB.	Reduce the file size or split the values into smaller files.
Too many rows in file	A file contains more than 10,000 data rows.	Split the values into smaller files.
File not found	The filename in the filter does not match an uploaded file.	Make sure the filename in your filter exactly matches the uploaded file name.
Invalid file format	The CSV header or JSON keys do not match the filter field.	Check that the CSV header matches the filter field name, or that the JSON is an array of objects.
Field not found	The field does not exist in the dataset.	Check field names with Get dataset metadata.

What does Filter return?

Filter returns a snapshot_id, the ID of the filtered snapshot the request created:

{ "snapshot_id": "s_m123456789" }

Use the snapshot_id with the snapshot endpoints once the job completes:

• Get snapshot metadata to check the job status.
• Get snapshot parts for large or split snapshots.
• Snapshot content to download the records.
• Deliver snapshot to send the records to a configured destination.

How much does Filter cost?

Filter costs $2.5 CPM (per 1,000 records returned), the same rate as the Marketplace. There is no charge when the filter returns 0 records.

What errors can Filter return?

Status	Meaning	What to do
400	Bad filter or params	Check field names with Get dataset metadata.
401	Bad or missing API key	Check your Bearer token.
402	Not enough funds	Top up your balance or reduce records_limit.
404	Unknown dataset_id	Confirm the dataset ID.
422	Filter matched 0 records	Loosen your filter or check field values.
429	Too many parallel jobs (max 25 running per dataset) or rate limit hit (5 requests/minute or 120 requests/hour)	Wait for running jobs to finish, then retry with backoff. To run fewer jobs, combine compatible filters into one broader job.

Related

• Dataset list to find available Marketplace datasets.
• Get dataset metadata to view dataset fields and types.
• Filter syntax reference for every operator, filter groups and nesting.
• Search dataset (sync) for real-time lookups.
• Dataset API overview

Authorizations

Authorization · string · header · required

Use your Bright Data API Key as a Bearer token in the Authorization header.

How to authenticate:

1. Obtain your API Key from the Bright Data account settings at https://brightdata.com/cp/setting/users
2. Include the API Key in the Authorization header of your requests
3. Format: Authorization: Bearer YOUR_API_KEY

Example:

Authorization: Bearer YOUR_API_KEY

Learn how to get your Bright Data API key: https://docs.brightdata.com/api-reference/authentication

Query Parameters

dataset_id · string

ID of the dataset to filter (required in multipart/form-data mode)

Example:

"gd_l1viktl72bvl7bjuj0"

records_limit · integer

Limit the number of records to be included in the snapshot

Example:

1000

Body

application/jsonmultipart/form-data

dataset_id · string · required

ID of the dataset to filter

Example:

"gd_l1viktl72bvl7bjuj0"

filter · Single field filter · object · required

Single field filter

Filters group

Single field filter w/out value

Show child attributes

Example:

{
  "name": "name",
  "operator": "=",
  "value": "John"
}

records_limit · integer

Limit the number of records to be included in the snapshot

Example:

1000

Response

200

application/json

Job of creating the snapshot successfully started

snapshot_id · string

ID of the snapshot