Crawl4ai (Web Scraping & Crawling)

🚀 Crawl4AI MCP Server

An implementation of the Model Context Protocol (MCP) server that integrates Crawl4AI into Cursor AI, providing web scraping and crawling capabilities as tools for LLMs in Cursor Composer's proxy mode.

🚀 Quick Start

This MCP server integrates Crawl4AI into Cursor AI, offering web scraping and crawling functions as tools for LLMs in Cursor Composer's proxy mode.

✨ Features

• Single-page scraping
• Website crawling

📦 Installation

Basic setup instructions are also provided in the Official Documentation: MCP Server Quick Start.

Set up your environment

First, install uv and set up your Python project and environment:

MacOS/Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Make sure to restart your terminal afterward so that the uv command is recognized.

Complete the following steps:

1. Clone the repository.
2. Install dependencies using UV (only the first time):

# Navigate to the crawl4ai-mcp directory
cd crawl4ai-mcp

# Install dependencies (only the first time)
uv venv
uv sync

# Activate the virtual environment
source .venv/bin/activate

# Run the server
python main.py

3. Add it to Cursor's MCP server or Claude's MCP server. You may need to provide the full path to the uv executable in the command field. You can obtain it by running which uv on MacOS/Linux or where uv on Windows.

{
  "mcpServers": {
    "Crawl4AI": {
      "command": "uv",
      "args": [
        "--directory",
        "/ABSOLUTE/PATH/TO/PARENT/FOLDER/crawl4ai-mcp",
        "run",
        "main.py"
      ]
    }
  }
}

💻 Usage Examples

This MCP server provides the following tools for LLMs:

Basic Usage

1. scrape_webpage(url: str)

   • Description: Use Crawl4AI to scrape the content and metadata of a specified webpage.
   • Parameters:
     • url (string, required): The URL of the webpage to scrape.
   • Returns: A list containing a TextContent object with the scraped content in Markdown format as JSON.

2. crawl_website(url: str, crawl_depth: int = 1, max_pages: int = 5)

   • Description: Use Crawl4AI to start a crawling activity from a specified URL, with the depth and number of pages limited as required.
   • Parameters:
     • url (string, required): The starting URL for the crawl.
     • crawl_depth (integer, optional, default value is 1): The maximum crawling depth relative to the starting URL.
     • max_pages (integer, optional, default value is 5): The maximum number of pages to scrape during the crawl.
   • Returns: A list containing a TextContent object with a JSON array result representing the crawled page information (including URL, success status, Markdown content, or error).

🔧 Technical Details

System Requirements

Python 3.10 or higher installed.

```markdown