How to set up Bright Data with Dify

Add Bright Data’s web scraping plugin to Dify to extract data from 50+ platforms including Amazon, LinkedIn and Instagram, plus search and markdown.

Building an AI startup?

You might be eligible for our Startup Program. Get fully funded access to the infrastructure you’re reading about right now (up to $20K value).

Learn more

The Bright Data plugin for Dify pulls structured web data, search results and page markdown into your Dify workflows. Install it from the Dify Marketplace, add your Bright Data API key, and drop the scraping tools into any workflow. It covers 50+ platforms including Amazon, LinkedIn, Instagram and YouTube.

For the latest plugin updates, see the Bright Data Dify plugin repository.

Prerequisites

• A Bright Data account with an API key
• A Dify account with access to Dify Studio

How to add the Bright Data plugin to Dify

1. Install the plugin

   Install the Bright Data plugin from the Dify Marketplace.

2. Get your Bright Data API key

   • Log in to your Bright Data dashboard.
   • Go to Account settings.
   • Generate an API key if you do not have one.
3. Add a scraping tool to a workflow

   1. Go to Dify Studio and open a Workflow.
   2. Add one of the Bright Data tools:
      • Structured Data Feeds: extract structured data from 50+ platforms
      • Scrape As Markdown: convert any webpage to clean markdown
      • Search Engine: get search results from Google, Bing or Yandex
   3. Enter your Bright Data API key when prompted.
   4. Connect an LLM node to process or summarize the scraped data.
4. Run an example workflow

   The banner image above shows this workflow. To extract Amazon product information and summarize it:

   1. START: input a product URL
   2. Structured Data Feeds: extract product details
   3. LLM: summarize into readable text
   4. END: output a clean product summary

   Two tips for reliable workflows:

   • Reference every stage to the output of the previous stage
   • Set a high character limit on input fields (for the URL field, choose the “short paragraph” var option)

Available tools

The plugin adds three tools to Dify.

Structured data feeds

Extract structured data from popular platforms:

• E-commerce: Amazon, eBay, Walmart, Best Buy, Etsy, Zara
• Social media: Instagram, Facebook, TikTok, YouTube, X (Twitter)
• Professional: LinkedIn profiles, companies, jobs
• Business: Crunchbase, ZoomInfo
• Maps and reviews: Google Maps, booking sites
• News: news sources and articles

Scrape as markdown

Convert any webpage into clean, readable markdown for:

• Content analysis
• Documentation extraction
• Article processing

Search across engines

Get search results from major search engines:

• Google
• Bing
• Yandex and others

Use cases

• E-commerce monitoring: track product prices and availability
• Lead generation: extract business information from LinkedIn
• Content research: gather articles and news for analysis
• Market research: monitor competitor websites and social media
• SEO analysis: track search engine results and rankings

Advanced: use Bright Data MCP

The Dify plugin uses hosted APIs. For advanced workflows, you can also integrate the Bright Data MCP server, which exposes the full Bright Data scraping and automation toolset over the Model Context Protocol.

Call MCP tools from Dify using custom HTTP requests or external service nodes to add capabilities like browser automation and real-time scraping.

• Explore MCP on GitHub
• Try MCP in the Smithery Playground