AI Web Scrapers Documentation

This document provides an overview of the AI-powered web scrapers integrated into our system. These scrapers interact with advanced AI models to retrieve structured responses to user prompts, including relevant metadata such as sources, citations, and contextual information.

Overview

Our platform supports the following AI scrapers:

• ChatGPT
• Perplexity
• Gemini
• Grok
• Google - AI Mode
• Copilot

Each scraper is designed to handle user prompts and return enriched responses, with varying levels of detail and supporting data.

How AI Scraping Differs from Traditional SERP Scraping

AI bot scraping offers a smarter, more efficient alternative to traditional SERP (Search Engine Results Page) scraping. While SERP scraping involves extracting raw HTML from search engine result pages—often requiring complex parsing and frequent maintenance—AI bot scraping uses natural language models to understand the intent behind a query and return structured, context-rich answers. Key advantages of AI bot scraping over SERP scraping:

• ✅ Understands query context and delivers direct answers
• ✅ Includes citations, sources, and enriched content (e.g., maps, product links)
• ✅ More resilient to UI changes and search engine updates
• ✅ Easier to integrate via official APIs
• ✅ Provides a better user experience with cleaner, more relevant results

In contrast, SERP scraping is limited to surface-level data, often lacks context, and may violate search engine terms of service. AI bot scraping is the modern, scalable solution for intelligent data extraction.

---

Scraper	Short Description	Includes
ChatGPT Scraper	Generates detailed, conversational answers using OpenAI’s ChatGPT.	Hyperlinks Citations (when available) Product recommendations Embedded map data Available countries
Perplexity Scraper	Provides concise, well-sourced answers via Perplexity AI, ideal for research.	Hyperlinked citations for verification Available countries
Copilot Scraper	Offers contextual answers with sources using Microsoft’s Copilot	Hyperlinked citations Contextual explanations Available countries
Gemini Scraper	Delivers informative responses with potential web searches via Google’s Gemini.	Web-based citations and sources Hyperlinked references (when available) Country targeting (European countries not available)
Google AI Scraper	Collect Google AI mode search results by prompt	Available countries
Grok Scraper	Collect Grok search results by prompt	Hyperlinked citations Available countries

Web search control varies by scraper. ChatGPT and Grok support a web_search input that allows or disables live web search, and a web_search_triggered output that confirms whether a search actually ran. See Query fan-out and web search control for full details, plus the availability matrix for Google AI Mode and other LLM Scrapers. Gemini does not currently expose a web_search control — the decision is made internally by Gemini and prompt phrasing may influence whether a web search is triggered.