Mendableai Firecrawl MCP Server

🚀 Fire Salamander Web Crawler System

The Fire Salamander Web Crawler System is a powerful tool designed for efficient web data collection. It offers multiple core components, a comprehensive logging system, and robust error - handling mechanisms, enabling users to easily perform web crawling tasks.

🚀 Quick Start

📦 Installation

# Install the Fire Crawl MCP server using npm
npm install fire - crawl - mcp - server

# Clone the source code repository (optional)
git clone https://github.com/your - repository.git
cd your - repository

Configuration File Example (config.json)

{
  "fireCrawl": {
    "apiKey": "your_api_key_here", // Required: Fire Salamander API key
    "proxySettings": { // Optional: Proxy configuration
      "enabled": false, // Whether to enable the proxy
      "host": "", // Proxy server address
      "port": "" // Proxy server port
    },
    "logging": { // Logging settings
      "level": "INFO", // Log level: DEBUG, INFO, WARNING, ERROR, CRITICAL
      "outputFile": "./fire_crawl.log" // Log file path
    }
  }
}

✨ Features

💻 Web Crawler Tool

• Name: Fire Salamander Crawler
• Function: Initiate web page crawling from a specified URL, supporting multi - threading acceleration and depth - first crawling.
• Configuration Parameters:
  • url: Required, the starting crawling address.
  • maxDepth: Optional, default is 3, the maximum crawling depth.
  • limit: Optional, default is 50, the upper limit of the number of web pages to be crawled.

Batch Processing Tool

• Name: Fire Salamander Batch Processing
• Function: Support submitting multiple crawling tasks at once and view the processing status through a unique ID.
• Operation Process:
  1. Submit a batch processing task.
  2. Obtain the operation ID.
  3. Regularly query the processing progress.

Status Check Tool

• Name: Fire Salamander Status Check
• Function: View the execution status and results of the current batch processing task through the provided operation ID.

🔧 Technical Details

Logging System

• Log Levels:
  • DEBUG: Record detailed debugging information.
  • INFO: System running status.
  • WARNING: Warnings that may affect functionality.
  • ERROR: Records of known errors.
  • CRITICAL: Serious problems that may cause service interruption.

Error Handling Mechanism

• Retry Strategy:
  • Automatically wait and retry when rate - limiting is detected.
  • Maximum number of retries: 5 times.
  • Retry interval: Randomly between 2 seconds and 10 seconds.

📚 Documentation

Development Guide

# Install project dependencies
npm install

# Build the production package
npm run build

# Run unit tests
npm test

Contribution Method

1. Fork the code repository.
2. Create a feature branch.
3. Submit code modifications.
4. Submit a Pull Request.

📄 License

• License Type: MIT License
• Copyright Information: © 2023 Fire Salamander Team