Firecrawl MCP Server

🚀 Firecrawl System

Firecrawl is a powerful web crawler tool that supports deep web page crawling, intelligent data extraction, and automated task execution. It features a modular architecture, offering high scalability and flexibility, making it suitable for various complex web application scenarios.

✨ Features

1. Web Crawl & Monitoring

• Deep Web Crawling: Supports multi-threaded concurrent crawling with flexible configuration to adapt to different website structures.
• Web Page Change Detection: Continuously monitors target web pages to detect content updates or changes promptly.
• Network Traffic Analysis: Monitors network status in real-time to identify abnormal traffic and potential security threats.

2. Data Processing & Analysis

• Data Cleaning: Automatically removes redundant information to improve data quality.
• Pattern Recognition: Uses machine learning algorithms to discover hidden data associations and trends.
• Log Mining: Parses system logs to extract valuable information and behavior patterns.

3. Automation & Task Execution

• Scheduled Task Scheduling: Configures periodic tasks to automatically perform crawling, analysis, and other operations.
• Rule Engine: Triggers automated operations based on preset conditions for intelligent business process management.
• Exception Handling Mechanism: Automatically retries or adjusts strategies when encountering network errors or anti-crawling mechanisms.

📦 Installation

HTTP Request Settings

{
  "proxy": {
    "enabled": true,
    "host": "127.0.0.1",
    "port": 8080
  },
  "headers": {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.113 Safari/537.3",
    "Accept-Language": "zh-CN,zh;q=0.9"
  },
  "timeout": {
    "connect": 10000,
    "read": 10000
  }
}

Data Storage Settings

{
  "database": {
    "type": "mysql",
    "host": "localhost",
    "port": 3306,
    "username": "root",
    "password": "password"
  },
  "filesystem": {
    "enabled": true,
    "directory": "/var/www/html/data/",
    "gzip": true
  }
}

Logging Settings

{
  "log_level": "INFO",
  "log_file": "/var/log/firecrawl.log",
  "console_logging": true,
  "file_logging": true,
  "retention_days": 30
}

💻 Usage Examples

Basic Usage

const FireCrawler = require('firecrawl');

async function basicCrawl() {
    try {
        const crawler = new FireCrawler();
        const response = await crawler.goto('http://example.com');
        console.log(response.status);
        console.log(response.html);
    } catch (error) {
        console.error('An error occurred during crawling:', error.message);
    }
}

basicCrawl();

Advanced Usage

Deep Web Crawling

const FireCrawler = require('firecrawl');

async function deepCrawl() {
    try {
        const crawler = new FireCrawler();
        await crawler.start({
            startUrls: ['http://example.com/'],
            maxDepth: 5,
            concurrency: 10
        });
        console.log('Crawling completed. Number of pages retrieved:', crawler.totalPages);
    } catch (error) {
        console.error('An error occurred during deep crawling:', error.message);
    }
}

deepCrawl();

Content Extraction

const FireCrawler = require('firecrawl');

async function contentExtraction() {
    try {
        const crawler = new FireCrawler();
        const response = await crawler.goto('http://example.com');
        const extractedData = crawler.extract({
            rules: [
                { selector: 'h1', attribute: 'text' },
                { selector: '.price', attribute: 'text' }
            ]
        });
        console.log('Extracted data:', extractedData);
    } catch (error) {
        console.error('An error occurred during content extraction:', error.message);
    }
}

contentExtraction();

📚 Documentation

System Log Examples

Runtime Logs

INFO: System started successfully, version: 1.0.0
INFO: Starting crawler task, target URL: http://example.com
DEBUG: Current number of threads: 4
WARNING: Anti-crawling mechanism detected, adjusting strategy...
ERROR: Unable to connect to the target server, retrying...
SUCCESS: Page crawled successfully, status code: 200
INFO: Crawler task completed, total pages crawled: 100

Error Logs

ERROR: Timeout error occurred during the request
ERROR: Invalid HTTP response header information
FATAL: Database connection failed, unable to save data

Network Traffic Analysis Examples

Basic Statistics

Response Time Distribution

Response Time (ms) Range | Percentage
-------------------------|-----------
< 100 | 70%
100 - 200 | 25%
> 200 | 5%

Development Guide

Installation & Configuration

1. Install dependencies:

   npm install firecrawl
   

2. Initialize the configuration file:

   firecraw init
   

3. Start the service:

   node app.js
   

Extension Development

• Plugin Mechanism: The system supports custom plugins, allowing developers to extend functional modules.
• Hook Functions: Provides various hook interfaces for inserting custom logic at specific stages.
• Data Processors: Allows for customized data processing workflows to meet different business needs.

Error Handling & Debugging

Common Errors and Solutions

1. Connection Timeout
   • Check network configuration.
   • Adjust timeout settings.

   crawler.config.timeout = 30000;
   

2. Anti-crawling Mechanism Triggered
   • Modify request header information.
   • Use a proxy server.

   crawler.useProxy();
   

3. Database Connection Failure
   • Check the database status.
   • Confirm credential information.

   db.connect().catch(err => console.error('Database connection error:', err));
   

Troubleshooting

1. System Fails to Start
   • Check if dependencies are installed correctly.
   • View startup logs to find specific error information.
2. Incomplete Data Extraction
   • Confirm if extraction rules are configured correctly.
   • Simulate requests to check if the page structure has changed.
3. Performance Bottlenecks
   • Analyze system load to identify bottleneck areas.
   • Optimize database query efficiency.
   • Increase server resources.

Performance Tuning

1. Thread Pool Configuration

   crawler.config.concurrency = 50;
   

2. Cache Strategy Adjustment

   crawler.useCache(true);
   

3. Database Optimization
   • Index optimization.
   • Query optimizer tuning.

Security Considerations

1. Prevent DDoS Attacks
   • Limit concurrent connections.
   • Implement rate limiting.
2. Data Protection
   • Encrypt data during transmission.
   • Regularly back up data.
3. Permission Management
   • Minimize user permissions.
   • Regularly review access logs.

Release Notes

Version 1.0.0 (2023-10-10)

• Initial version released, including basic functional modules.
• Provides a web interface and command-line tools.

Version 1.1.0 (2023-10-25)

• New network layer monitoring function added.
• Data extraction performance optimized, with a 30% efficiency improvement.
• Several known serious bugs fixed.

Version 1.2.0 (2023-11-10)

• Supports multiple database backends (MySQL, PostgreSQL).
• Log analysis module added.
• System stability improved, reducing the crash rate.

Contact Information

• Official Website: https://firecrawl.com
• Email Support: support@fire

🔧 Technical Details

System Architecture Diagram

+-------------------+       +-------------------+
|                   |       |                   |
|     User          |       |     Database      |
|     (Web UI)      |       |     (MySQL/PostgreSQL)|
|                   |       |                   |
+-------------------+       +-------------------+
          ||                         ||
          ||                         ||
+--------+||+---------+        +-----+||+-------+
|  Web   |++|  Network  |        |  Log  |++|  Database|
| Server |   |  Layer    |        |  Storage |     |
+--------+   +---------+        +-----+
          ||                         ||
          ||                         ||
+--------+||+---------+        +-----+||+-------+
|  Controller |++|  Request  |        |  Monitoring |++|  Analysis |
|  Module  |   |  Handling |        |  System  |     |  Module |
+--------+   +---------+        +-----+

Data Flow Diagram

Start -> Parse Configuration -> Initialize Components -> Execute Task -> Process Data -> Store Results -> End

📄 License

The Firecrawl system is released under the MIT open-source license, allowing users to freely use, modify, and distribute the software. Please refer to the LICENSE file provided with the software for detailed license terms.