Crawl4ai

🚀 Crawl4AI MCP Server

Crawl4AI MCP Server is a high - performance web crawler system designed specifically for AI assistants. It supports connections via the Model Context Protocol (MCP) and offers a series of powerful tools to assist AI assistants in web page crawling and data processing.

🚀 Quick Start

Documentation Links

• Project Documentation
• Model Context Protocol Specification

✨ Features

Core Features

1. Web Crawling: Supports depth - first or breadth - first web page crawling starting from a specified URL.
2. Data Storage: Stores the crawled content in a distributed database to ensure high availability and persistence.
3. Content Extraction: Uses advanced natural language processing techniques to extract key information from documents.

Advanced Features

1. Task Management:
   • crawl: Initiates a web crawling operation starting from a specified URL.
   • getCrawl: Retrieves crawled data based on the ID.
   • listCrawls: Lists all crawl records or filters the results by domain name.
2. Search Function: Supports full - text retrieval in the crawled documents based on query keywords.
3. Structured Extraction: Extracts structured data, such as tables and lists, from a specified URL.

📚 Documentation

Project Structure

src/
├── main.ts: Entry file to start the server and related services
├── crawler/
│   ├── crawler.ts: Core logic of the web crawler
│   └── storage.ts: Data storage and retrieval module
├── extractor/
│   └── extractor.ts: Implementation of the content extraction algorithm
└── tools/
    ├── crawlTool.ts: Definition of the crawler tool
    ├── searchTool.ts: Definition of the search tool
    └── extractTool.ts: Definition of the extraction tool

📦 Installation

Local Installation

npm install

Runtime Environment Preparation

• Ensure that the Node.js version is 16.0.0 or higher.
• Install Docker and configure the relevant environment.

Running and Deployment

Local Startup

npm run dev

Docker Deployment

docker-compose up -d

💻 Usage Examples

Authentication Mechanism

• Implements OAuth - based secure authentication.
• Supports authentication using API keys.

Connecting with MCP Clients

1. Add the server address as a tool data source in Claude Desktop or other MCP clients.

Configuration Options

Flexible configuration can be achieved by modifying environment variables:

MAX_CRAWL_DEPTH=3         # Default crawl depth limit
MAX_CRAWL_PAGES=100       # Default limit on the number of crawled pages
API_VERSION=v1            # API version number
OAUTH_CLIENT_ID=your_id    # OAuth client ID
OAUTH_CLIENT_SECRET=your_secret  # OAuth client secret

🔧 Technical Details

Current Development Stage

1. Project Initialization: Completed the configuration of Cloudflare Worker and the basic setup of TypeScript.
2. MCP Server Implementation: Completed the adaptation of the MCP protocol and the service framework.
3. Crawl4AI Component Integration: Integrated the Crawl4AI functional modules.

Future Plans

1. Performance Optimization:
   • Improve the efficiency of the crawler and optimize the storage structure.
2. Function Expansion:
   • Support multi - threaded crawling.
   • Add more data processing functions, such as image recognition and table parsing.

🤝 Contribution Guidelines

Community contributions are welcome! Please submit issues or Pull Requests via GitHub to participate in the project development. For detailed rules, please refer to CONTRIBUTING.md.

📚 Appendix

Frequently Asked Questions

• How to improve the crawler efficiency? You can optimize the crawling strategy and add proxy servers.
• What are the limitations of data storage? Currently, it supports mainstream databases, and more storage solutions will be expanded in the future.

Contact Information

• Email: support@crawl4ai.com
• GitHub: Crawl4AI Project

Thank you for using the Crawl4AI MCP Server! If you need help, please feel free to contact us.