Mcma123 Firecrawl MCP Server

🚀 FireCrawl Project Documentation

FireCrawl is an open - source web crawling and information extraction tool. It offers various functional modules, including web page scraping, content parsing, and search optimization. It is suitable for a wide range of web data acquisition tasks.

🚀 Quick Start

📦 Installation

# Clone the repository
git clone https://github.com/yourusername/firecrawl.git

# Navigate to the project directory
cd firecrawl

# Install dependencies
npm install

# Start the service
node index.js

✨ Features

1. Web Page Crawling Tool (firecrawler_crawl)

Launch an asynchronous crawler to fetch the content of the target web page.

{
  "name": "firecrawler_crawl",
  "arguments": {
    "url": "https://example.com", // The URL address to be crawled
    "maxDepth": 2, // Maximum crawling depth
    "limit": 100, // Maximum number of links to be fetched per page
    "allowExternalLinks": false, // Whether to allow crawling external links
    "deduplicateSimilarUrls": true // Whether to remove similar URLs
  }
}

2. Content Extraction Tool (firecrawler_extract)

Leverage a large language model (LLM) to extract structured information from web pages.

{
  "name": "firecrawler_extract",
  "arguments": {
    "urls": ["https://example.com/page1", "https://example.com/page2"], // List of URLs to be extracted
    "prompt": "Extract product information including name, price, and description", // Extraction prompt statement
    "systemPrompt": "You are a helpful assistant that extracts product information", // System prompt
    "schema": { // Structured data schema
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "price": { "type": "number" },
        "description": { "type": "string" }
      },
      "required": ["name", "price"]
    },
    "allowExternalLinks": false, // Whether to allow extracting information from external links
    "enableWebSearch": false, // Whether to enable the web search function
    "includeSubdomains": false // Whether to include sub - domains
  }
}

Extraction Tool Options

• urls: List of URLs to be extracted
• prompt: Custom prompt statement
• systemPrompt: System prompt
• schema: JSON schema for structured data extraction
• allowExternalLinks: Whether to allow extracting information from external links
• enableWebSearch: Whether to enable the web search function to obtain more context information
• includeSubdomains: Whether to include sub - domains

3. Search Tool (firecrawler_search)

Perform a search task on the web and optionally extract content from the search results.

{
  "name": "firecrawler_search",
  "arguments": {
    "query": "your search query", // Search keywords
    "limit": 5, // Result quantity limit
    "lang": "en", // Search language
    "country": "us", // Search region country code
    "scrapeOptions": { // Content extraction options
      "formats": ["markdown"], // Extraction format
      "onlyMainContent": true // Whether to extract only the main web page content
    }
  }
}

4. Logging and Monitoring

FireCrawl provides detailed logging functionality, making it easy for users to monitor the running status of the crawler. Log files are stored in the ./logs directory by default.

# View real - time logs
tail -f ./logs/crawler.log

5. Error Handling

The system has a built - in error capture mechanism that automatically records problems encountered during the crawling process and provides error codes and brief descriptions.

Common Error Codes

• 403 Forbidden: The target website refuses access.
• 404 Not Found: The requested resource does not exist.
• 503 Service Unavailable: The server is temporarily unavailable.
• CONNECTION_TIMEOUT: Connection timeout.
• PARSE_ERROR: Content parsing failed.

🔧 Configuration Options

Network Request Configuration

{
  "requestOptions": {
    "headers": { // Request header information
      "User - Agent": "FireCrawler/1.0", // Simulate browser access
      "Accept - Language": "zh - CN" // Accept Chinese content
    },
    "timeout": 30000, // Single request timeout (milliseconds)
    "proxy": null // Optional, set proxy server information
  }
}

Storage Configuration

{
  "storageOptions": {
    "type": "filesystem", // Supported storage types: filesystem, database, s3
    "path": "./data", // File system storage path
    "dbConfig": null, // Database connection information (if using a database)
    "awsS3Config": null // AWS S3 storage configuration (if using cloud storage)
  }
}

📚 Logging System

FireCrawl provides detailed logging functionality and supports multiple logging levels:

• DEBUG: Debug information
• INFO: Normal running information
• WARNING: Warning information that may affect functionality
• ERROR: Logs when an error occurs
• CRITICAL: Logs for severe errors that cause service interruption

Log Format

The default log format is as follows:

[TIME] - [LEVEL] - [MODULE]: Message

🔧 Error Handling Mechanism

The system has a built - in error capture module that supports custom error handlers. By default, it records error information and pauses the crawler.

Custom Error Handling Example

module.exports.errorHandler = function(err) {
  console.error('Custom error handler:', err);
  // Optional: Add additional error handling logic
};

🛠️ Development and Contribution

Install Dependencies

git clone https://github.com/yourusername/firecrawl.git
cd firecrawl
npm install

Start the Service

node index.js

Code Submission Guidelines

1. Write clear commit messages for each commit.
2. Ensure that unit tests pass.
3. Update relevant documentation.

🤝 Contribution Guidelines

Welcome contributions from the community! Please follow these steps:

1. Fork the project repository.
2. Develop on your own branch.
3. Ensure that all test cases pass before submitting a Pull Request.
4. Add updated documentation and comments.

📄 License

FireCrawl is licensed under the MIT license. You are welcome to use and modify the code freely.