Firecrawl MCP Server 6kk

🚀 FireCrawl

FireCrawl is an intelligent web crawler and content processing platform based on large language models. It integrates advanced AI technology with efficient web data scraping capabilities to help users acquire, parse, and analyze information from the internet.

🚀 Quick Start

FireCrawl combines advanced AI technology with efficient web data scraping capabilities, enabling users to acquire, parse, and analyze information from the internet.

✨ Features

• Intelligent Crawling: Combines AI understanding with traditional crawler technology to achieve efficient and intelligent web page scraping.
• Content Parsing: Utilizes large language models to conduct in - depth understanding and semantic analysis of the scraped content.
• Automated Processing: Offers a one - stop solution from data collection to result generation.

📦 Installation

# Install dependencies
npm install

# Build the project
npm run build

# Run tests
npm test

💻 Usage Examples

1. Web Crawling Tool (firecrawler)

# Example usage
python firecrawler.py --url https://example.com --depth 2

Key Features:

• Supports multi - threading/multi - processing acceleration.
• Intelligent anti - anti - crawler mechanism.
• Flexible scraping strategy configuration.

2. Text Parsing Tool (llm_parser)

# Example usage
python llm_parser.py --file parsed.html --model gpt - 4

Functional Features:

• Multi - language support.
• Document understanding and structuring.
• Natural language processing.

3. Deep Network Research Tool (firecrawler_deep_search)

{
  "name": "firecrawler_deep_search",
  "arguments": {
    "query": "What are the basic principles of quantum computing?", 
    "maxDepth": 3,
    "timeLimit": 60,
    "maxUrls": 50
  }
}

Parameter Description:

• query: Required, search keyword or question.
• maxDepth: Optional, default is 2, crawling depth limit.
• timeLimit: Optional, default is 30 seconds, time limit.
• maxUrls: Optional, default is 50, upper limit of the number of scraped URLs.

Return Result:

{
  "summary": "Quantum computing is a parallel computing method based on qubits...", 
  "sources": ["https://quantum.com/1", ...],
  "resultsCount": 3456
}

🔧 Technical Details

Logging System

Log Recording

• Operation status and progress.
• Performance metrics.
• Quota usage.
• Rate limit tracking.
• Error conditions.

Example Log Information:

[INFO] FireCrawl service has been successfully started.
[WARNING] API quota is about to be exhausted.
[ERROR] Network request timed out, retrying...

Error Handling Mechanism

Handling Strategies:

• Automatically retry temporary errors.
• Intellectually bypass rate limits.
• Provide detailed error information reports.
• Issue quota usage warnings.
• Adapt to network fluctuations.

Example Error Response:

{
  "content": [
    {
      "type": "text",
      "text": "Error: API quota has been exhausted, please wait for 30 seconds and try again..."
    }
  ],
  "isError": true
}

🤝 Contribution

1. Fork the repository.
2. Create a feature branch.
3. Run tests: npm test.
4. Submit a Pull Request.

🙏 Acknowledgments

Thanks to the following contributors:

• @vrknetha
• @cawstudios

Thanks to MCP.so and Klavis AI for their support, and @zhangwei for his guidance.

📄 License

This project is licensed under the MIT License. Please refer to the license file for detailed information.

For more information about FireCrawl, please visit the official website or follow our Github repository.