MCP HubMCP Hub
mendableai

firecrawl-mcp-server

by: mendableai

Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.

1821created 06/12/2024
Visit
Firecrawl
scraping

📌Overview

Purpose: Firecrawl MCP Server is designed to provide a robust implementation of the Model Context Protocol (MCP) for enhanced web scraping functionalities integrated with Firecrawl.

Overview: The Firecrawl MCP Server offers advanced web scraping capabilities, including support for JavaScript rendering, URL discovery, and deep research. It efficiently manages tasks with rate limiting, automatic retries, and credit usage monitoring, making it suitable for both cloud and self-hosted environments.

Key Features:

  • Comprehensive Scraping Tools: Includes advanced tools for scraping, batch processing, and deep research, enabling users to extract data from single or multiple URLs effortlessly.

  • Robust Error Handling: Implements automatic retries with exponential backoff for transient errors and features comprehensive logging for performance metrics and credit tracking.

  • Customizable Configuration: Offers a range of configurable environment variables to manage API keys, retries, and credit monitoring thresholds, providing flexibility for different deployment scenarios.

Firecrawl MCP Server

A Model Context Protocol (MCP) server implementation that integrates with Firecrawl for web scraping capabilities.

Features

  • Scrape, crawl, search, extract, deep research, and batch scrape support
  • Web scraping with JS rendering
  • URL discovery and crawling
  • Automatic retries with exponential backoff
  • Efficient batch processing with built-in rate limiting
  • Comprehensive logging system
  • Support for cloud and self-hosted FireCrawl instances
  • Mobile/Desktop viewport support
  • Smart content filtering with tag inclusion/exclusion

Installation

Running with npx

env FIRECRAWL_API_KEY=fc-YOUR_API_KEY npx -y firecrawl-mcp

Manual Installation

npm install -g firecrawl-mcp

Running on Cursor

Requires Cursor version 0.45.6+.

  1. Open Cursor Settings
  2. Go to Features > MCP Servers
  3. Click "+ Add New MCP Server"
  4. Enter the following:
    • Name: "firecrawl-mcp"
    • Type: "command"
    • Command: env FIRECRAWL_API_KEY=your-api-key npx -y firecrawl-mcp

Running on Windsurf

Add this to your ./codeium/windsurf/model_config.json:

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }
}

Installing via Smithery (Legacy)

To install FireCrawl for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @mendableai/mcp-server-firecrawl --client claude

Configuration

Environment Variables

Required for Cloud API

  • FIRECRAWL_API_KEY: Required when using cloud API.
  • FIRECRAWL_API_URL (Optional): Custom API endpoint for self-hosted instances.

Optional Configuration

  • FIRECRAWL_RETRY_MAX_ATTEMPTS: Maximum number of retry attempts (default: 3).
  • FIRECRAWL_RETRY_INITIAL_DELAY: Delay before first retry (default: 1000 ms).
  • FIRECRAWL_RETRY_MAX_DELAY: Maximum delay between retries (default: 10000 ms).
  • FIRECRAWL_RETRY_BACKOFF_FACTOR: Exponential backoff multiplier (default: 2).
  • FIRECRAWL_CREDIT_WARNING_THRESHOLD: Credit usage warning threshold (default: 1000).
  • FIRECRAWL_CREDIT_CRITICAL_THRESHOLD: Critical credit usage threshold (default: 100).

Usage with Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "mcp-server-firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "YOUR_API_KEY_HERE",
        "FIRECRAWL_RETRY_MAX_ATTEMPTS": "5",
        "FIRECRAWL_RETRY_INITIAL_DELAY": "2000",
        "FIRECRAWL_RETRY_MAX_DELAY": "30000",
        "FIRECRAWL_RETRY_BACKOFF_FACTOR": "3",
        "FIRECRAWL_CREDIT_WARNING_THRESHOLD": "2000",
        "FIRECRAWL_CREDIT_CRITICAL_THRESHOLD": "500"
      }
    }
  }
}

Rate Limiting and Batch Processing

  • Automatic rate limit handling with exponential backoff.
  • Efficient parallel processing for batch operations.
  • Automatic retries for transient errors.

Available Tools

1. Scrape Tool (firecrawl_scrape)

Scrape content from a single URL with advanced options.

{
  "name": "firecrawl_scrape",
  "arguments": {
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true,
    "waitFor": 1000,
    "timeout": 30000,
    "mobile": false,
    "includeTags": ["article", "main"],
    "excludeTags": ["nav", "footer"],
    "skipTlsVerification": false
  }
}

2. Batch Scrape Tool (firecrawl_batch_scrape)

Scrape multiple URLs efficiently.

{
  "name": "firecrawl_batch_scrape",
  "arguments": {
    "urls": ["https://example1.com", "https://example2.com"],
    "options": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

3. Check Batch Status (firecrawl_check_batch_status)

Check the status of a batch operation.

{
  "name": "firecrawl_check_batch_status",
  "arguments": {
    "id": "batch_1"
  }
}

4. Search Tool (firecrawl_search)

Search the web and extract content from results.

{
  "name": "firecrawl_search",
  "arguments": {
    "query": "your search query",
    "limit": 5,
    "lang": "en",
    "country": "us",
    "scrapeOptions": {
      "formats": ["markdown"],
      "onlyMainContent": true
    }
  }
}

5. Crawl Tool (firecrawl_crawl)

Start an asynchronous crawl with advanced options.

{
  "name": "firecrawl_crawl",
  "arguments": {
    "url": "https://example.com",
    "maxDepth": 2,
    "limit": 100,
    "allowExternalLinks": false,
    "deduplicateSimilarURLs": true
  }
}

6. Extract Tool (firecrawl_extract)

Extract structured information from web pages.

{
  "name": "firecrawl_extract",
  "arguments": {
    "urls": ["https://example.com/page1", "https://example.com/page2"],
    "prompt": "Extract product information including name, price, and description",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    },
    "allowExternalLinks": false
  }
}

7. Deep Research Tool (firecrawl_deep_research)

Conduct deep web research on a query.

{
  "name": "firecrawl_deep_research",
  "arguments": {
    "query": "how does carbon capture technology work?",
    "maxDepth": 3,
    "maxUrls": 50
  }
}

8. Generate LLMs.txt Tool (firecrawl_generate_llmstxt)

Generate a standardized llms.txt for a given domain.

{
  "name": "firecrawl_generate_llmstxt",
  "arguments": {
    "url": "https://example.com",
    "maxUrls": 20,
    "showFullText": true
  }
}

Logging System

The server includes comprehensive logging for:

  • Operation status and progress
  • Performance metrics
  • Credit usage monitoring

Error Handling

The server provides robust error handling with:

  • Automatic retries for transient errors
  • Detailed error messages
  • Credit usage warnings

Development

# Install dependencies
npm install

# Build
npm run build

# Run tests
npm test

Contributing

  1. Fork the repository
  2. Create your feature branch
  3. Run tests: npm test
  4. Submit a pull request

License

MIT License - see LICENSE file for details.