Crawl4ai MCP Server Rln

🚀 Crawl4ai MCP Server

A MCP server that provides web crawling functionality, using crawl4ai and outputting in Markdown format for large language models.

🚀 Quick Start

This Crawl4ai MCP Server offers web - crawling capabilities. It utilizes crawl4ai and presents the results in Markdown format, making it suitable for large language models.

📦 Installation

Prerequisites

• Node.js
• Access to a crawl4ai instance: https://docs.crawl4ai.com/core/docker-deployment/

Setup

1. Clone the repository:

git clone https://github.com/Kirill812/crawl4ai-mcp-server.git
cd crawl4ai-mcp-server

2. Install dependencies:

npm install

3. Build the server:

npm run build

4. Add the server configuration to your environment variables:

{
  "mcpServers": {
    "crawl4ai": {
      "command": "node",
      "args": [
        "/path/to/crawl4ai-mcp-server/build/index.js"
      ],
      "env": {
        "CRAWL4AI_API_URL": "http://127.0.0.1:11235",  // Optional: URL of the crawl4ai API service
        "CRAWL4AI_AUTH_TOKEN": "your-auth-token"       // Optional: Authentication token if required
      }
    }
  }
}

Replace the environment variables with your own values:

• CRAWL4AI_API_URL: URL of the crawl4ai API service (Optional)
• CRAWL4AI_AUTH_TOKEN: Authentication token for the API (Optional)

✨ Features

Tools

• crawl_urls - Crawl web pages and get Markdown content with references
  • Parameters:
    • urls (Required): A list of URLs to crawl

Response Format

The tool returns the Markdown content of each URL along with its references. Multiple URLs are separated by a horizontal line (---). Example:

This is the content of the first URL [^1]

[^1]: https://example.com

---

This is the content of the second URL [^2]

[^2]: https://example.org

💻 Usage Examples

Basic Usage

# Install and build the server according to the installation steps

Advanced Usage

# Use automatic rebuild for development
npm run watch

🔧 Technical Details

Error Handling

Common issues and solutions:

1. Ensure that the URLs are valid and accessible.
2. If authentication is required, make sure the token is valid.
3. Check the network connection to the crawl4ai API service.
4. In case of timeout errors, try reducing the number of URLs in each request.
5. If blocked by a website, the service will automatically retry and use different user - agents.

📄 License

This MCP server is distributed under the MIT License. This means you are free to use, modify, and distribute the software, subject to the terms and conditions of the MIT License. For more details, see the LICENSE file in the project repository.