Crawl4ai MCP Server

🚀 Crawl4AI MCP Server

This is an intelligent information acquisition server based on MCP (Model Context Protocol), providing powerful search capabilities and web content understanding functions optimized for large language models to AI assistant systems. Through multi - engine search and intelligent content extraction, it helps AI systems efficiently acquire and understand Internet information and supports output in markdown_with_citations format, facilitating citation and processing by large language models.

🚀 Quick Start

Get started with the Crawl4AI MCP Server by following the installation and usage instructions below.

✨ Features

• Multi - engine Search: By default, it integrates the DuckDuckGo search engine and supports the extension of Google search (configuration required).
• LLM - optimized Content Extraction: A content filtering and extraction mechanism designed specifically for large language models, automatically removing useless information such as navigation bars and advertisements while retaining key content and URL references.
• Focus on Information Value: Filters invalid fragments through a minimum word count threshold (10) to ensure the high quality and high relevance of the output content.

📦 Installation

Regular Installation

# Create and activate a virtual environment
python -m venv env
source env/bin/activate  # On macOS/Linux
env\Scripts\activate      # On Windows

# Install dependencies
pip install -r requirements.txt

# Configure the server
cp config_demo.json config.json

# Start the server
python src/index.py

Claude Client Plugin Installation

# Install to Claude Sonnet 3.5 using the smithery CLI
smithery plugin install @weidwonder/crawl4ai-mcp-server --development

💻 Usage Examples

Search Function (search)

Example Request:

curl -X POST "http://localhost:8000/api/search" \
     -H "Content-Type: application/json" \
     -d '{"query":"AI technology development history","gl":null,"hl":null}'

Output Format:

• Supports two formats: markdown_with_citations and plaintext, with markdown_with_citations as the default.
• Each result includes a title, content excerpt, original URL, and relevance score.

Content Extraction and Processing (read_url)

Example Request:

curl -X POST "http://localhost:8000/api/read_url" \
     -H "Content-Type: application/json" \
     -d '{"url":"https://example.com"}'

📚 Documentation

Project Structure

crawl4ai_mcp_server/
├── src/
│   ├── index.py      # Main server implementation
│   └── search.py     # Search function implementation
├── config_demo.json  # Configuration file example
├── pyproject.toml    # Project configuration
├── requirements.txt  # Dependency list
└── README.md         # Project documentation

Configuration Instructions

1. Copy the configuration template:

   cp config_demo.json config.json
   

2. Configure Google search (optional):

   • Add the following content to config.json:

     {
         "google": {
             "api_key": "your_google_api_key",
             "cse_id": "your_google_cse_id"
         }
     }
     

🔄 Update Log

• February 8, 2025: Added a search function, supporting DuckDuckGo (default) and Google search.
• February 7, 2025: Refactored the project structure, implemented with FastMCP, and optimized dependency management.
• February 7, 2025: Optimized content filtering configuration, improving token efficiency and maintaining URL integrity.

📄 License

This project is licensed under the MIT License.

🤝 Contribution Guidelines

Welcome to submit issues and Pull Requests! GitHub repository: https://github.com/weidwonder/crawl4ai-mcp-server

🧑‍💻 Author Information

• Project Lead: weidwonder
• Main Developer: Claude Sonnet 3.5
  • The code was completely written by Claude. It took approximately 3 hours in total, including 0.5 hours for coding, 0.5 hours for environment preparation, and 2 hours for debugging.

🙏 Acknowledgments

Thanks to all developers who have contributed to the project!

Special thanks to:

• The Crawl4ai project for providing excellent web content extraction technology support.