📌Overview

Purpose: To provide a production-ready Model Context Protocol (MCP) server that facilitates AI-powered web scraping through integration with the ScapeGraph API.

Overview: The ScrapeGraph MCP Server empowers language models to utilize advanced scraping capabilities, ensuring enterprise-grade reliability and efficient data extraction from the web.

Key Features:

markdownify(website_url: str): Converts any webpage into clean and structured markdown format for easier readability and data manipulation.
smartscraper(user_prompt: str, website_url: str): Utilizes AI to extract structured data from any webpage based on user-defined prompts, enhancing data retrieval accuracy.
searchscraper(user_prompt: str): Performs AI-driven web searches that return structured and actionable results, promoting effective information gathering.

ScrapeGraph MCP Server

A production-ready Model Context Protocol (MCP) server that provides seamless integration with the ScapeGraph AI API. This server enables language models to leverage advanced AI-powered web scraping capabilities with enterprise-grade reliability.

Available Tools

The server provides the following enterprise-ready tools:

markdownify(website_url: str): Transform any webpage into clean, structured markdown format
smartscraper(user_prompt: str, website_url: str): Leverage AI to extract structured data from any webpage
searchscraper(user_prompt: str): Execute AI-powered web searches with structured, actionable results

Setup Instructions

To utilize this server, you'll need a ScapeGraph API key. Follow these steps to obtain one:

Navigate to the ScapeGraph Dashboard: https://dashboard.scrapegraphai.com
Create an account and generate your API key

Automated Installation via Smithery

For automated installation of the ScrapeGraph API Integration Server using Smithery:

npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude

Claude Desktop Configuration

Update your Claude Desktop configuration file with the following settings (located on the top right of the Cursor page):

(Remember to add your API key inside the config)

{
    "mcpServers": {
        "@ScrapeGraphAI-scrapegraph-mcp": {
            "command": "npx",
            "args": [
                "-y",
                "@smithery/cli@latest",
                "run",
                "@ScrapeGraphAI/scrapegraph-mcp",
                "--config",
                "\"{\\\"scrapegraphApiKey\\\":\\\"YOUR-SGAI-API-KEY\\\"}\""
            ]
        }
    }
}

The configuration file is located at:

Windows: %APPDATA%/Claude/claude_desktop_config.json
macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

Cursor Integration

Add the ScrapeGraphAI MCP server on the settings.

Example Use Cases

The server enables sophisticated queries such as:

Analyze and extract the main features of the ScapeGraph API
Generate a structured markdown version of the ScapeGraph homepage
Extract and analyze pricing information from the ScapeGraph website
Research and summarize recent developments in AI-powered web scraping
Create a comprehensive summary of the Python documentation website

Error Handling

Robust error handling is implemented with detailed, actionable error messages for:

API authentication issues
Malformed URL structures
Network connectivity failures
Rate limiting and quota management

Common Issues

Windows-Specific Connection

When running on Windows systems, you may need to use the following command to connect to the MCP server:

C:\Windows\System32\cmd.exe /c npx -y @smithery/cli@latest run @ScrapeGraphAI/scrapegraph-mcp --config "{\"scrapegraphApiKey\":\"YOUR-SGAI-API-KEY\"}"

This ensures proper execution in the Windows environment.

License

This project is distributed under the MIT License.

Acknowledgments

Special thanks to tomekkorbak (https://github.com/tomekkorbak) for his implementation of oura-mcp-server (https://github.com/tomekkorbak/oura-mcp-server), which served as the starting point for this repo.

Made with ❤️ by the ScrapeGraphAI Team