How to scrape Google data in bulk

Trigger large-scale Google scraping jobs, monitor progress and retrieve results using the Bright Data async /trigger endpoint. Returns up to 1 GB per response.

This guide shows you how to scrape Google data at scale using the asynchronous /trigger endpoint. Use this when you have more than 20 inputs, need discovery by keyword, location or input filters, or want delivery to a webhook or S3.

Not sure whether to use sync or async? Read Understanding sync vs. async requests.

Prerequisites

• A Bright Data account with an active API key
• Familiarity with the synchronous request flow

Step 1: Trigger the collection

Send a POST request to the /trigger endpoint with your input array. This example collects five Google Maps places in a single batch:

cURL

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_m8ebnr0q2qlklc02fz&format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    {"url": "https://www.google.com/maps/place/Empire+State+Building"},
    {"url": "https://www.google.com/maps/place/Central+Park"},
    {"url": "https://www.google.com/maps/place/Times+Square"},
    {"url": "https://www.google.com/maps/place/Statue+of+Liberty"},
    {"url": "https://www.google.com/maps/place/Brooklyn+Bridge"}
  ]'

import requests

response = requests.post(
    "https://api.brightdata.com/datasets/v3/trigger",
    params={"dataset_id": "gd_m8ebnr0q2qlklc02fz", "format": "json"},
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json=[
        {"url": "https://www.google.com/maps/place/Empire+State+Building"},
        {"url": "https://www.google.com/maps/place/Central+Park"},
        {"url": "https://www.google.com/maps/place/Times+Square"},
        {"url": "https://www.google.com/maps/place/Statue+of+Liberty"},
        {"url": "https://www.google.com/maps/place/Brooklyn+Bridge"},
    ],
)

snapshot = response.json()
print("Snapshot ID:", snapshot["snapshot_id"])

const response = await fetch(
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_m8ebnr0q2qlklc02fz&format=json",
  {
    method: "POST",
    headers: {
      "Authorization": "Bearer YOUR_API_KEY",
      "Content-Type": "application/json",
    },
    body: JSON.stringify([
      { url: "https://www.google.com/maps/place/Empire+State+Building" },
      { url: "https://www.google.com/maps/place/Central+Park" },
      { url: "https://www.google.com/maps/place/Times+Square" },
      { url: "https://www.google.com/maps/place/Statue+of+Liberty" },
      { url: "https://www.google.com/maps/place/Brooklyn+Bridge" },
    ]),
  }
);

const snapshot = await response.json();
console.log("Snapshot ID:", snapshot.snapshot_id);

You should see a 200 response with a snapshot_id:

{
  "snapshot_id": "s_m1a2b3c4d5e6f7g8h"
}

Save this ID. You need it to check progress and download results.

Discovery with async

Discovery is the primary reason to use async. Most Google discovery modes can return large result sets, and some require structured inputs that aren’t URLs.

Discover Google Maps places by location:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_m8ebnr0q2qlklc02fz&format=json&type=discover_new&discover_by=location" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"country": "US", "lat": 40.7484, "long": -73.9857, "zoom_level": 14, "keyword": "coffee shop"}]'

Discover Google Shopping products by keyword:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_ltppk50q18kdw67omz&format=json&type=discover_new&discover_by=keyword" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"keyword": "wireless headphones", "country": "US"}]'

Discover Google Flights by input filters:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_mhng7wen1rw0a3gvpf&format=json&type=discover_new&discover_by=input_filters" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{
    "origin": "JFK",
    "destination": "LAX",
    "departure": "2026-06-15",
    "return": "2026-06-22",
    "trip_type": "round_trip",
    "adults": 1,
    "cabin": "economy"
  }]'

Discover Google Hotels by search:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_mg3gjfmg12tc2n5d4d&format=json&type=discover_new&discover_by=search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{
    "search_term": "New York",
    "check_in_date": "2026-06-15",
    "check_out_date": "2026-06-18",
    "guest_number": 2,
    "country": "US",
    "currency": "USD"
  }]'

Step 2: Monitor progress

Poll the snapshot status until it shows ready. This takes 30 seconds to several minutes depending on input count and whether discovery is involved.

cURL

curl "https://api.brightdata.com/datasets/v3/progress/s_m1a2b3c4d5e6f7g8h" \
  -H "Authorization: Bearer YOUR_API_KEY"

import time

snapshot_id = "s_m1a2b3c4d5e6f7g8h"

while True:
    status_response = requests.get(
        f"https://api.brightdata.com/datasets/v3/progress/{snapshot_id}",
        headers={"Authorization": "Bearer YOUR_API_KEY"},
    )
    status = status_response.json().get("status")
    print(f"Status: {status}")

    if status == "ready":
        break
    time.sleep(10)

const snapshotId = "s_m1a2b3c4d5e6f7g8h";

let status = "collecting";
while (status !== "ready") {
  const statusResponse = await fetch(
    `https://api.brightdata.com/datasets/v3/progress/${snapshotId}`,
    { headers: { "Authorization": "Bearer YOUR_API_KEY" } }
  );
  const statusData = await statusResponse.json();
  status = statusData.status;
  console.log("Status:", status);

  if (status !== "ready") {
    await new Promise((r) => setTimeout(r, 10000));
  }
}

Status values:

Status	Meaning
collecting	Scraping is in progress
digesting	Data is being processed
ready	Results are available for download
failed	The collection encountered an error

Step 3: Download results

Once the status is ready, download the scraped data:

cURL

curl "https://api.brightdata.com/datasets/v3/snapshot/s_m1a2b3c4d5e6f7g8h?format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -o results.json

results_response = requests.get(
    f"https://api.brightdata.com/datasets/v3/snapshot/{snapshot_id}",
    params={"format": "json"},
    headers={"Authorization": "Bearer YOUR_API_KEY"},
)

results = results_response.json()
print(f"Collected {len(results)} records")

const resultsResponse = await fetch(
  `https://api.brightdata.com/datasets/v3/snapshot/${snapshotId}?format=json`,
  { headers: { "Authorization": "Bearer YOUR_API_KEY" } }
);

const results = await resultsResponse.json();
console.log(`Collected ${results.length} records`);

You’ve successfully triggered, monitored and downloaded a batch Google scraping job. See the Download Snapshot API reference for format options and part-by-part download flow.

Skip polling with webhooks

If you don’t want to poll for status, add an endpoint parameter to receive results automatically:

curl -X POST \
  "https://api.brightdata.com/datasets/v3/trigger?dataset_id=gd_m8ebnr0q2qlklc02fz&format=json&endpoint=https://your-server.com/webhook" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://www.google.com/maps/place/Empire+State+Building"}]'

See Webhook delivery for the full setup.

Limits and constraints

Constraint	Value
Max inputs per async request	5,000
Max input file size	1 GB
Max concurrent batch requests	100
Max concurrent single-input requests	1,500
Webhook delivery size	Up to 1 GB
API download size	Up to 5 GB

Troubleshooting

Getting a 429 Too Many Requests error?

You’ve exceeded the concurrent request limit. Reduce the number of parallel requests or combine inputs into fewer, larger batches. Each batch can include up to 1 GB of input data.

Snapshot status shows 'failed'?

Check that all input URLs are valid and correctly formatted for the target Google product. Review the error details in the snapshot response or in the Logs tab of your Bright Data dashboard.

Results are incomplete or missing some records?

Individual inputs can fail while the overall job succeeds. Check the snapshot response for any errors field and retry failed inputs in a separate request.

Discovery returned zero results?

Verify your discovery parameters. Location-based Maps discovery needs valid lat/long and a realistic zoom level (12 to 16). Flights discovery needs IATA codes and future dates. Hotels discovery needs a valid destination string.

Next steps

Webhook delivery

Push results to your HTTP endpoint automatically.

API reference

Full endpoint specs, parameters and response schemas.