Spider MCP

🚀 Spider MCP - Web Search Crawler Service

A web search MCP service based on pure crawler technology, built with Node.js.

✨ Features

• ❌ No Official API Required: Completely based on crawler technology, without relying on third - party official APIs.
• 🔍 Intelligent Search: Supports Bing web and news search.
• 📰 News Search: Built - in news search with time filtering.
• 🕷️ Pure Crawler: No official API dependency, uses Puppeteer for web scraping.
• 🚀 High Performance: Supports batch web scraping.
• 📊 Health Monitoring: Complete health check and metrics monitoring.
• 📝 Structured Logging: Uses Winston for structured logs.
• 🔒 Anti - Detection: Supports User - Agent rotation and other anti - bot measures.
• 🔗 Smart URL Cleaning: Automatically cleans promotional parameters while preserving essential information.

📦 Installation

1. Install dependencies

npm install

or use pnpm

pnpm install

2. Download Puppeteer browser

npx puppeteer browsers install chrome

3. Environment configuration

Copy and configure the environment variables file:

cp .env.example .env

Edit the .env file according to your needs.

🚀 Quick Start

4. Start the service

• Development mode:

npm run dev

• Production mode:

npm start

The service will start at http://localhost:3000.

💻 Usage Examples

web_search

Unified search tool supporting both web and news search:

• Web Search: searchType: "web"
• News Search: searchType: "news" with time filtering
• Note: searchType is a required parameter and must be explicitly specified

Usage Examples:

# Web search
Use web_search tool to search "Node.js tutorial" with searchType set to web, return 10 results

# News search
Use web_search tool to search "tech news" with searchType set to news, return 5 results from past 24 hours

Other Tools

• get_webpage_content: Get webpage content and convert to specified format.
• get_webpage_source: Get raw HTML source code of webpage.
• batch_webpage_scrape: Batch scrape multiple webpages.

📚 Documentation

MCP Configuration

Chatbox Configuration

Create mcp-config.json file in Chatbox:

{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["src/mcp/server.js"],
      "env": {
        "NODE_ENV": "production"
      },
      "description": "Spider MCP - Web search and webpage scraping tools",
      "capabilities": {
        "tools": {}
      }
    }
  }
}

Other MCP Clients

{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["path/to/spider-mcp/src/mcp/server.js"]
    }
  }
}

Tech Stack

• Node.js (>= 18.0.0)
• Express.js - Web framework
• Puppeteer - Browser automation
• Cheerio - HTML parsing
• Axios - HTTP client
• Winston - Logging
• @modelcontextprotocol/sdk - MCP protocol support

Development

Project Structure

spider-mcp/
├── src/
│   ├── index.js          # Main entry file
│   ├── mcp/
│   │   └── server.js     # MCP server
│   ├── routes/           # Route definitions
│   │   ├── search.js     # Search routes
│   │   └── health.js     # Health check routes
│   ├── services/         # Business logic
│   │   └── searchService.js # Search service
│   └── utils/            # Utility functions
│       └── logger.js     # Logging utility
├── logs/                 # Log files directory
├── tests/                # Test files
├── package.json          # Project configuration
├── .env.example          # Environment variables example
├── mcp-config.json       # MCP configuration example
└── README.md             # Project documentation

⚠️ Important Note

1. Anti - bot Measures: This service uses various techniques to avoid detection, but still needs to comply with robots.txt and terms of use.
2. Rate Limiting: It's recommended to control request frequency reasonably to avoid putting pressure on target websites.
3. Legal Compliance: Please ensure compliance with local laws and website terms of use when using this service.
4. Resource Consumption: Puppeteer will start Chrome browser, please pay attention to memory and CPU usage.
5. URL Cleaning: Automatically cleans promotional parameters but may affect some special link functionality.

📄 License

MIT License

💪 Contributing

Issues and Pull Requests are welcome!