Web Crawler MCP Server

🚀 Web Crawler MCP Server

A Model Context Protocol (MCP) server that offers a web crawling and content extraction tool, suitable for AI assistants like Claude Desktop, Cursor, and other MCP-compatible clients.

🚀 Quick Start

The Web Crawler MCP Server is a powerful tool for web crawling and content extraction. It can be easily integrated with MCP-compatible clients such as Claude Desktop and Cursor.

✨ Features

• Extract and clean the main text content from any public web page.
• Use Puppeteer with the _stealth plugin to bypass anti-bot protection.
• Return readable, whitespace-normalized text for consumption by LLMs.
• Easily integrate with Claude Desktop and other MCP-compatible clients.

📦 Installation

Prerequisites

• Node.js (v16 or higher)
• MCP-compatible clients (e.g., Claude Desktop, Cursor)
• (Optional) Puppeteer dependencies for some Linux environments

Steps

1. Install dependencies:

npm install

2. Build the server:

npm run build

💻 Usage Examples

Basic Usage

You can directly run the server:

node build/index.js

Or configure it as an MCP server in your client (e.g., Claude Desktop):

{
  "mcpServers": {
    "web-crawler-mcp": {
      "command": "node",
      "args": ["<absolute-path-to>/server/web_crawler/build/index.js"]
    }
  }
}

Available Tools

Web Crawler

• Description: Extract and return the cleaned text content of the specified URL.
• Input:
  • url (string, required): The URL from which to extract content.

Example

{
  "tool_name": "web-crawler",
  "arguments": {
    "url": "https://openai.com/news"
  }
}

🔧 Technical Details

Development Commands

• npm run build — Compile TypeScript to JavaScript.
• npm run watch — Monitor and rebuild changes in real-time.
• npm run inspector — Start the MCP inspector for debugging.

Notes

• The server starts a real browser instance (headless: false) for optimal compatibility.
• The output is plain text, suitable for LLM input.
• For advanced parsing, modify the Cheerio logic in src/index.ts.

📄 License

MIT