Crawl4ai MCP

🚀 Crawl4AI Intelligent Web Crawler and AI Processing Server (MCP)

Crawl4AI is an MCP (Model - Controller - Processor) server designed for intelligent web crawling and AI - based content analysis. It offers easy - to - use APIs, enabling data scraping from websites and content processing using the Claude AI model.

🎯 Target Users

Crawl4AI primarily serves individuals and organizations that need in - depth analysis of specific website content. Unlike general search engines or AI assistants that provide broad but shallow results, Crawl4AI offers deep insights into specific target websites.

Ideal Use Cases:

• Researchers: Extract structured information from specific websites or academic resources.
• Content Creators: Analyze competitors' content or industry trends.
• Data Analysts: Process web data for business intelligence purposes.
• Developers: Build applications that require web content analysis capabilities.
• Digital Marketers: Analyze industry websites, blogs, or competitors' content.
• Business Analysts: Collect industry - specific information from multiple sources.
• Knowledge Workers: Conduct comprehensive analysis of content in specific domains.

Benefits for Users

Compared with general search and AI tools, Crawl4AI has significant advantages:

• Depth over Breadth: Focus on comprehensive analysis of specific websites rather than broad but superficial results.
• Customizable Crawling Parameters: Have full control over crawling depth, content extraction methods, and processing workflows.
• Programmatic Integration: Easily integrate web content analysis capabilities into applications and data pipelines.
• Flexible AI Processing: Apply different analysis methods (summarization, fact extraction, classification, etc.) to the same content.
• Efficient Content Understanding: Extract insights from text through advanced language models.
• Enhanced Decision - Making Support: Provide reliable data support for business and technical decisions.

✨ Features

• Web Crawling: Utilize Puppeteer and Cheerio for efficient page scraping and parsing.
• Content Processing: Analyze and understand scraped content using advanced AI models.
• Scalable Architecture: Modular design, supporting flexible expansion and customization of functions.
• Logging and Monitoring: Track the running status in detail through Winston.
• Security and Compliance: Built - in anti - anti - scraping mechanisms to ensure compliance with website usage policies.

📦 Installation

1. Clone the project repository:

git clone https://github.com/yourusername/crawl4ai.git
cd crawl4ai

2. Install dependencies:

npm install

3. Configure environment variables:

   • Set PORT as the port on which the server runs. The default value is 3000.
   • Set ANTHROPIC_API_KEY as the API key you obtained from Anthropic.

4. Start the service:

npm start

💻 Usage Examples

Basic Usage

curl -X POST http://localhost:3000/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "depth": 1,
    "aiProcessing": {
      "task": "summarize"
    }
  }'

Advanced Usage

Use Case 1: Scrape and Generate a Web Page Summary

curl -X POST http://localhost:3000/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/post",
    "depth": 2,
    "aiProcessing": {
      "task": "summarize",
      "options": {
        "length": "short"
      }
    }
  }'

Use Case 2: Scrape and Identify Web Page Themes

curl -X POST http://localhost:3000/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product",
    "depth": 1,
    "aiProcessing": {
      "task": "classify",
      "model": "content_classifier"
    }
  }'

📚 Configuration Options

📄 License

This project is licensed under the MIT License.

🙏 Acknowledgments

This project uses the following excellent libraries:

• Express: An efficient web application framework.
• Puppeteer: A powerful browser automation tool.
• Cheerio: A fast HTML manipulation library.
• Winston: A flexible logging solution.
• @anthropic-ai/sdk: Anthropic's AI development toolkit.