Getting started

There are two ways to use Crawl API:

API-Based collection
No-Code collection (via Control Panel)

How to collect via API

Trigger a data collection job via a simple HTTP POST
Specify the URLs and output format
Receive a snapshot_id to retrieve results later.

Code Example

curl -H "Authorization: Bearer API_KEY" -H "Content-Type: application/json" -d '[{"url":"https://example.com"},{"url":"https://example.com/1"}]' "https://api.brightdata.com/datasets/v3/trigger?dataset_id=<dataset_id>&include_errors=<true/false>&custom_output_fields=<custom_output_fields>"

Query Parameters:

query

required

Your dataset ID (e.g., gd_m6gjtfmeh43we6cqc)

query

default:"true"

include error logs in results

query

markdown, html, ld_json, etc. \Choose the format that best fits your workflow:

# Main Article Title

This is the introduction paragraph with **bold text** and *italics*.

## Subheading

- List item one
- List item two

> This is a blockquote from the articlesh Code Example

[Link text](https://example.com/more-info)
![Image description](https://example.com/image.jpg)

Main Article Title

This is the introduction paragraph with bold text and italics.

Subheading

* List item one
* List item two

This is a blockquote from the articlesh Code Example

Link text [https://example.com/more-info]
[Image: Image description]

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Example Article</title>
  <link rel="stylesheet" href="/styles.css">
  <script src="/analytics.js"></script>
</head>
<body>
  <header>
    <nav><!-- Navigation HTML --></nav>
  </header>
  <main>
    <h1>Main Article Title</h1>
    <p>This is the introduction paragraph with <strong>bold text</strong> and <em>italics</em>.</p>
    <h2>Subheading</h2>
    <ul>
      <li>List item one</li>
      <li>List item two</li>
    </ul>
    <blockquote>This is a blockquote from the article</blockquote>
    <a href="https://example.com/more-info">Link text</a>
    <img src="https://example.com/image.jpg" alt="Image description">sh Code Example
  </main>
</body>
  <footer><!-- Footer HTML --></footer>
</html>

{
  "@context":"https://schema.org",
  "@type":"Article",
  "headline":"Main Article Title",
  "author":{
    "@type":"Person",
    "name":"Jane Smith"
  },
  "datePublished":"2023-08-15T12:00:00Z",
  "dateModified":"2023-08-15T14:22:31Z",
  "publisher":{
    "@type":"Organization",
    "name":"Example Publication",
    "logo":{
      "@type":"ImageObject",
      "url":"https://example.com/logo.png"
    }
  },
  "image":"https://example.com/image.jpg",
  "description":"This is the introduction paragraph with bold text and italics.",
  "mainEntityOfPage":"https://example.com/article"
}

How results are delivered

Deliver results to:

Webhooks
External storage (S3, GCS, etc.)
Direct download via API or Control Panel

No-Code Scraper (Control Panel)

Use our Control Panel to launch crawls without writing a single line of code. Steps:

Open the Crawl API Control Panel
Enter the target domain or URLs
Choose your output format
Start the crawl
Download results directly from the dashboard

Introduction

Product Guides

How to collect via API

Query Parameters:

How results are delivered

No-Code Scraper (Control Panel)

​How to collect via API

​Query Parameters:

​How results are delivered

​No-Code Scraper (Control Panel)

How to collect via API

Query Parameters:

How results are delivered

No-Code Scraper (Control Panel)