Crawlab MCP Server

🚀 Crawler Laboratory MCP Server

This is a Model Context Protocol (MCP) server designed for the Crawler Laboratory (Crawlab), which allows AI applications to interact with Crawlab's functionalities.

🚀 Quick Start

The MCP server provides a standard way for AI applications to access Crawlab's features, including:

• Spider management (Create, Read, Update, Delete)
• Task management (Run, Cancel, Restart)
• File management (Read, Write)
• Resource access (Spiders, Tasks)

✨ Features

Architecture

The MCP Server/Client architecture facilitates communication between AI applications and Crawlab:

graph TB
    User[User] --> Client[MCP Client]
    Client --> LLM[LLM Provider]
    Client <--> Server[MCP Server]
    Server <--> Crawlab[Crawlab API]

    subgraph "MCP System"
        Client
        Server
    end

    subgraph "Crawlab System"
        Crawlab
        DB[(Database)]
        Crawlab <--> DB
    end

    class User,LLM,Crawlab,DB external;
    class Client,Server internal;

    %% Flow comments
    LLM -.-> |Tool call| Client
    Client -.-> |Execute tool call| Server
    Server -.-> |API request| Crawlab
    Crawlab -.-> |API response| Server
    Server -.-> |Tool result| Client
    Client -.-> |Human-readable response| User

    classDef external fill:#f9f9f9,stroke:#333,stroke-width:1px;
    classDef internal fill:#d9edf7,stroke:#31708f,stroke-width:1px;

Communication Process

1. User Query: The user sends a natural language query to the MCP client.
2. LLM Processing: The client forwards the query to an LLM provider (e.g., Claude, OpenAI).
3. Tool Selection: The LLM identifies the intent and invokes the spider creation tool.
4. Spider Creation: The MCP server sends a command to the Crawlab API to create a spider.
5. Spider Creation Completed: The spider is successfully created, and detailed information is returned to the user.

📦 Installation

Windows System

Install using pip

pip install mcp-server-windows

Manual installation steps

1. Download the latest Windows version of the MCP server.
2. Extract the files to the desired location.
3. Open the command prompt, navigate to the extracted directory, and run:

python server.py

Linux System

Install using pip

pip install mcp-server-linux

Manual installation steps

1. Download the latest Linux version of the MCP server.
2. Extract the files to the desired location.
3. Open the terminal, navigate to the extracted directory, and run:

python server.py

💻 Usage Examples

Basic Usage

Create a Spider

# Create a new spider with default configuration
python create_spider.py --name my_spider

# Create a spider with a configuration folder for a specific project
python create_spider.py --name my_project --project project_name

Run a Spider

# Run a single spider
python run_spider.py --spider my_spider

# Run a spider on all available nodes
python run_spider.py --spider my_spider --all-nodes

# Set arguments for the crawl command
python run_spider.py --spider my_spider --crawl-args arg1=value1,arg2=value2

Update Spider Code

# Update the content of a specific file
python update_spider_code.py --spider my_spider --file main.py --content "New Python code..."

# Update the code of all spiders
python update_spider_code.py --all --content "New Python code..."

Advanced Usage

There are no advanced usage examples provided in the original text.

📚 Documentation

Available Commands

Resource Access

• spiders: List all spiders
• tasks: List all tasks

Tool Usage

Spider Management

• get_spider: Get detailed information about a specific spider
• create_spider: Create a new spider
• update_spider: Update an existing spider
• delete_spider: Delete a spider

Task Management

• get_task: Get detailed information about a specific task
• run_spider: Run a spider
• cancel_task: Cancel a running task
• restart_task: Restart a task
• get_task_logs: Get task logs