Anycrawl MCP Server

🚀 AnyCrawl MCP Server

AnyCrawl MCP Server is a powerful tool for web scraping and crawling, designed for Cursor, Claude, and other LLM clients through the Model Context Protocol (MCP). It enables efficient extraction of web content and seamless integration with various applications.

✨ Features

• Web Scraping: Extract content from single URLs in multiple output formats.
• Website Crawling: Crawl entire websites with configurable depth and limits.
• Search Engine Integration: Search the web and optionally scrape results.
• Multiple Engines: Support for Playwright, Cheerio, and Puppeteer.
• Flexible Output: Markdown, HTML, text, screenshots, and structured JSON.
• Async Operations: Non-blocking crawl jobs with status monitoring.
• Error Handling: Robust error handling and logging.
• Multiple Modes: STDIO (default), MCP(HTTP), SSE; cloud-ready with Nginx proxy.

📦 Installation

Running with npx

ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp

Manual installation

npm install -g anycrawl-mcp-server

ANYCRAWL_API_KEY=YOUR-API-KEY anycrawl-mcp

📚 Configuration

Set the required environment variable:

export ANYCRAWL_API_KEY="your-api-key-here"

Optionally set a custom base URL:

export ANYCRAWL_BASE_URL="https://api.anycrawl.dev"  # Default

Get your API key

• Visit the AnyCrawl website and sign up or log in: AnyCrawl
• 🎉 Sign up for free to receive 1,500 credits — enough to crawl nearly 1,500 pages.
• Open the dashboard → API Keys → Copy your key.
• Copy the key and set it as the ANYCRAWL_API_KEY environment variable (see above).

💻 Usage

Available Modes

AnyCrawl MCP Server supports the following deployment modes: Default mode is STDIO (no env needed). Set ANYCRAWL_MODE to switch.

Quick Start Commands

# Development (local)
npm run dev            # STDIO (default)
npm run dev:mcp          # MCP mode (JSON /mcp)
npm run dev:sse          # SSE mode (/sse)

# Production (built output)
npm start              # STDIO (default)
npm run start:mcp
npm run start:sse

# Env examples
ANYCRAWL_MODE=MCP ANYCRAWL_API_KEY=YOUR-KEY npm run dev:mcp
ANYCRAWL_MODE=SSE ANYCRAWL_API_KEY=YOUR-KEY npm run dev:sse

Docker Compose (MCP + SSE with Nginx)

This repo ships a production-ready image that runs MCP (JSON) on port 3000 and SSE on port 3001 in the same container, fronted by Nginx. Nginx also supports API-key-prefixed paths /{API_KEY}/mcp and /{API_KEY}/sse and forwards the key via x-anycrawl-api-key header.

docker compose build
docker compose up -d

Environment variables used in Docker image:

• ANYCRAWL_MODE: MCP_AND_SSE (default in compose), or MCP, SSE
• ANYCRAWL_MCP_PORT: default 3000
• ANYCRAWL_SSE_PORT: default 3001
• CLOUD_SERVICE: true to extract API key from /{API_KEY}/... or headers
• ANYCRAWL_BASE_URL: default https://api.anycrawl.dev

Running on Cursor

Configuring Cursor. Note: Requires Cursor v0.45.6+.

For Cursor v0.48.6 and newer, add this to your MCP Servers settings:

{
  "mcpServers": {
    "anycrawl-mcp": {
      "command": "npx",
      "args": ["-y", "anycrawl-mcp"],
      "env": {
        "ANYCRAWL_API_KEY": "YOUR-API-KEY"
      }
    }
  }
}

For Cursor v0.45.6:

1. Open Cursor Settings → Features → MCP Servers → "+ Add New MCP Server"
2. Name: "anycrawl-mcp" (or your preferred name)
3. Type: "command"
4. Command:

env ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp

On Windows, if you encounter issues:

cmd /c "set ANYCRAWL_API_KEY=YOUR-API-KEY && npx -y anycrawl-mcp"

Running on VS Code

For manual installation, add this JSON to your User Settings (JSON) in VS Code (Command Palette → Preferences: Open User Settings (JSON)):

{
  "mcp": {
    "inputs": [
      {
        "type": "promptString",
        "id": "apiKey",
        "description": "AnyCrawl API Key",
        "password": true
      }
    ],
    "servers": {
      "anycrawl": {
        "command": "npx",
        "args": ["-y", "anycrawl-mcp"],
        "env": {
          "ANYCRAWL_API_KEY": "${input:apiKey}"
        }
      }
    }
  }
}

Optionally, place the following in .vscode/mcp.json in your workspace to share config:

{
  "inputs": [
    {
      "type": "promptString",
      "id": "apiKey",
      "description": "AnyCrawl API Key",
      "password": true
    }
  ],
  "servers": {
    "anycrawl": {
      "command": "npx",
      "args": ["-y", "anycrawl-mcp"],
      "env": {
        "ANYCRAWL_API_KEY": "${input:apiKey}"
      }
    }
  }
}

Running on Windsurf

Add this to ./codeium/windsurf/model_config.json:

{
  "mcpServers": {
    "mcp-server-anycrawl": {
      "command": "npx",
      "args": ["-y", "anycrawl-mcp"],
      "env": {
        "ANYCRAWL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Running with SSE Server Mode

The SSE (Server-Sent Events) mode provides a web-based interface for MCP communication, ideal for web applications, testing, and integration with web-based LLM clients.

Quick Start

# Development mode
ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp

# Or using npm scripts
ANYCRAWL_API_KEY=YOUR-API-KEY npm run dev:sse

Server Configuration

Optional server settings (defaults shown):

export ANYCRAWL_PORT=3000
export ANYCRAWL_HOST=0.0.0.0

Health Check

curl -s http://localhost:${ANYCRAWL_PORT:-3000}/health
# Response: ok

Generic MCP/SSE Client Configuration

For other MCP/SSE clients that support SSE transport, use this configuration:

{
  "mcpServers": {
    "anycrawl": {
      "type": "sse",
      "url": "https://mcp.anycrawl.dev/{API_KEY}/sse",
      "name": "AnyCrawl MCP Server",
      "description": "Web scraping and crawling tools"
    }
  }
}

or

{
  "mcpServers": {
    "AnyCrawl": {
      "type": "streamable_http",
      "url": "https://mcp.anycrawl.dev/{API_KEY}/mcp"
    }
  }
}

Environment Setup:

# Start SSE server with API key
ANYCRAWL_API_KEY=your-api-key-here npm run dev:sse

Cursor configuration for HTTP modes (streamable_http)

Configure Cursor to connect to your HTTP MCP server.

Local HTTP Streamable Server:

{
  "mcpServers": {
    "anycrawl-http-local": {
      "type": "streamable_http",
      "url": "http://127.0.0.1:3000/mcp"
    }
  }
}

Cloud HTTP Streamable Server:

{
  "mcpServers": {
    "anycrawl-http-cloud": {
      "type": "streamable_http",
      "url": "https://mcp.anycrawl.dev/{API_KEY}/mcp"
    }
  }
}

Note: For HTTP modes, set ANYCRAWL_API_KEY (and optional host/port) in the server process environment or in the URL. Cursor does not need your API key when using streamable_http.

💻 Usage Examples

Basic Usage

Here are some basic usage examples of the available tools:

Scrape Tool (anycrawl_scrape)

{
  "name": "anycrawl_scrape",
  "arguments": {
    "url": "https://example.com",
    "engine": "cheerio",
    "formats": ["markdown", "html"],
    "timeout": 30000
  }
}

Crawl Tool (anycrawl_crawl)

{
  "name": "anycrawl_crawl",
  "arguments": {
    "url": "https://example.com/blog",
    "engine": "playwright",
    "max_depth": 2,
    "limit": 50,
    "strategy": "same-domain",
    "poll_seconds": 3,
    "timeout_ms": 60000
  }
}

Advanced Usage

For more complex scenarios, you can combine different parameters and tools. For example, using the search tool to find relevant pages and then crawl them:

{
  "name": "anycrawl_search",
  "arguments": {
    "query": "latest AI research papers 2024",
    "engine": "google",
    "limit": 5,
    "scrape_options": {
      "engine": "cheerio",
      "formats": ["markdown"]
    }
  }
}

After getting the search results, you can use the crawl tool to further extract content from the relevant pages.

Output Formats

Markdown

Clean, structured markdown content perfect for LLM consumption.

HTML

Raw HTML content with all formatting preserved.

Text

Plain text content with minimal formatting.

Screenshot

Visual screenshot of the page.

Screenshot@fullPage

Full-page screenshot including content below the fold.

Raw HTML

Unprocessed HTML content.

JSON

Structured data extraction using custom schemas.

Engines

Cheerio

• Fast and lightweight
• Good for static content
• Server-side rendering

Playwright

• Full browser automation
• JavaScript rendering
• Best for dynamic content

Puppeteer

• Chrome/Chromium automation
• Good balance of features and performance

🔧 Error Handling

The server provides comprehensive error handling:

• Validation Errors: Invalid parameters or missing required fields
• API Errors: AnyCrawl API errors with detailed messages
• Network Errors: Connection and timeout issues
• Rate Limiting: Automatic retry with backoff

Logging

The server includes detailed logging:

• Debug: Detailed operation information
• Info: General operation status
• Warn: Non-critical issues
• Error: Critical errors and failures

Set log level with environment variable:

export LOG_LEVEL=debug  # debug, info, warn, error

Development

Prerequisites

• Node.js 18+
• npm

Setup

git clone <repository>
cd anycrawl-mcp
npm ci

Build

npm run build

Test

npm test

Lint

npm run lint

Format

npm run format

Contributing

1. Fork the repository
2. Create your feature branch
3. Run tests: npm test
4. Submit a pull request

📄 License

MIT License - see LICENSE file for details

Support

• GitHub Issues: Report bugs or request features
• Documentation: AnyCrawl API Docs
• Email: help@anycrawl.dev

About AnyCrawl

AnyCrawl is a powerful Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. It features native multi-threading for bulk processing and supports multiple output formats.

• Website: https://anycrawl.dev
• GitHub: https://github.com/any4ai/anycrawl
• API: https://api.anycrawl.dev