Crawlab

🚀 Crawler Laboratory MCP Server

This is a Model Context Protocol (MCP) server designed for Crawler Laboratory (Crawlab), allowing AI applications to interact with Crawlab's features.

🚀 Quick Start

The MCP server provides a standard way for AI applications to access Crawlab's features, including:

• Crawler management (Create, Read, Update, Delete)
• Task management (Run, Cancel, Restart)
• File management (Read, Write)
• Resource access (Crawlers, Tasks)

✨ Features

The MCP Server/Client architecture enables communication between AI applications and Crawlab:

graph TB
    User[User] --> Client[MCP Client]
    Client --> LLM[LLM Provider]
    Client <--> Server[MCP Server]
    Server <--> Crawlab[Crawlab API]

    subgraph "MCP System"
        Client
        Server
    end

    subgraph "Crawlab System"
        Crawlab
        DB[(Database)]
        Crawlab <--> DB
    end

    class User,LLM,Crawlab,DB External;
    class Client,Server Internal;

    %% Flow comments
    LLM -.-> |Tool call| Client
    Client -.-> |Execute tool call| Server
    Server -.-> |API request| Crawlab
    Crawlab -.-> |API response| Server
    Server -.-> |Tool result| Client
    Client -.-> |Human-readable response| User

    classDef External fill:#f9f9f9,stroke:#333,stroke-width:1px;
    classDef Internal fill:#d9edf7,stroke:#31708f,stroke-width:1px;

Communication Flow

1. User Query: The user sends a natural language query to the MCP client.
2. LLM Processing: The client forwards the query to an LLM provider (e.g., Claude, OpenAI).
3. Tool Selection: The LLM identifies the intent and invokes the crawler creation tool.
4. MCP Server Execution: The server sends commands to the Crawlab API.
5. Operation Completion: The crawler or task is executed, and the results are returned to the user.

📦 Installation

Install the MCP Server

pip install crawlab-mcp-server

Initialize the Database

crawlab init_db

Start the MCP Server

uvicorn --host 0.0.0.0 --port 8000 crawlab_mcp_server.app:app

💻 Usage Examples

Basic Usage

Create a Crawler

User: Create a new crawler named "Product Scraper" for an e-commerce project.
↓
LLM calls the create_spider tool.
↓
The MCP server sends a command to the Crawlab API.
↓
The crawler is created, and detailed information is returned to the user.

Run a Task

User: Run the "Product Scraper" crawler on all available nodes.
↓
LLM calls the run_spider tool.
↓
The MCP server sends a command to the Crawlab API.
↓
The task is started, and a confirmation is returned to the user.

Advanced Usage

You can interact with the system using natural language, for example:

• "List all my crawlers"
• "Create a new crawler and configure it according to these instructions..."
• "Show the code of crawler X"
• "Save this code to the main.py file of crawler X"
• "Run crawler X and notify me when it's done"
• "Show the results of the last run of crawler X"

📚 Documentation

Available Resources and Tools

The following are the underlying tools that support natural language interaction:

Resources

• spiders: List all crawlers
• tasks: List all tasks

Tools

Crawler Management

• get_spider: Get detailed information about a specific crawler
• create_spider: Create a new crawler
• update_spider: Update an existing crawler
• delete_spider: Delete a crawler

Task Management

• get_task: Get detailed information about a specific task
• run_spider: Run a crawler
• cancel_task: Cancel a running task
• restart_task: Restart a task
• get_task_logs: Get task logs