Spider MCP

🚀 Spider MCP - Web Search Crawler Service

A web search MCP service based on pure crawler technology, built with Node.js.

✨ Features

• ❌ No Official API Required: Completely based on crawler technology, without relying on third - party official APIs.
• 🔍 Intelligent Search: Supports Bing web and news search.
• 📰 News Search: Comes with built - in news search featuring time filtering.
• 🕷️ Pure Crawler: Does not depend on official APIs, and uses Puppeteer for web scraping.
• 🚀 High Performance: Supports batch web scraping.
• 📊 Health Monitoring: Equipped with complete health check and metrics monitoring.
• 📝 Structured Logging: Utilizes Winston for structured logging.
• 🔒 Anti - Detection: Supports User - Agent rotation and other anti - bot measures.
• 🔗 Smart URL Cleaning: Automatically cleans promotional parameters while retaining essential information.

📦 Installation

1. Install dependencies

npm install

or use pnpm

pnpm install

2. Download Puppeteer browser

npx puppeteer browsers install chrome

3. Environment configuration

Copy and configure the environment variables file:

cp .env.example .env

Edit the .env file according to your needs.

4. Start the service

Development mode:

npm run dev

Production mode:

npm start

The service will start at http://localhost:3000.

💻 Usage Examples

web_search

Unified search tool supporting both web and news search:

• Web Search: searchType: "web"
• News Search: searchType: "news" with time filtering
• Note: searchType is a required parameter and must be explicitly specified

Basic Usage

# Web search
Use web_search tool to search "Node.js tutorial" with searchType set to web, return 10 results

# News search
Use web_search tool to search "tech news" with searchType set to news, return 5 results from past 24 hours

Other Tools

• get_webpage_content: Get webpage content and convert to specified format.
• get_webpage_source: Get raw HTML source code of webpage.
• batch_webpage_scrape: Batch scrape multiple webpages.

📚 Documentation

MCP Configuration

Chatbox Configuration

Create mcp-config.json file in Chatbox:

{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["src/mcp/server.js"],
      "env": {
        "NODE_ENV": "production"
      },
      "description": "Spider MCP - Web search and webpage scraping tools",
      "capabilities": {
        "tools": {}
      }
    }
  }
}

Other MCP Clients

{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["path/to/spider-mcp/src/mcp/server.js"]
    }
  }
}

Important Notes

⚠️ Important Note

1. This service uses various techniques to avoid detection, but still needs to comply with robots.txt and terms of use.
2. It's recommended to control request frequency reasonably to avoid putting pressure on target websites.
3. Please ensure compliance with local laws and website terms of use when using this service.
4. Puppeteer will start Chrome browser, please pay attention to memory and CPU usage.
5. Automatically cleans promotional parameters but may affect some special link functionality.

Development

Project Structure

spider-mcp/
├── src/
│   ├── index.js          # Main entry file
│   ├── mcp/
│   │   └── server.js     # MCP server
│   ├── routes/           # Route definitions
│   │   ├── search.js     # Search routes
│   │   └── health.js     # Health check routes
│   ├── services/         # Business logic
│   │   └── searchService.js # Search service
│   └── utils/            # Utility functions
│       └── logger.js     # Logging utility
├── logs/                 # Log files directory
├── tests/                # Test files
├── package.json          # Project configuration
├── .env.example          # Environment variables example
├── mcp-config.json       # MCP configuration example
└── README.md             # Project documentation

🔧 Technical Details

Tech Stack

📄 License

This project is under the MIT License.

Contributing

Issues and Pull Requests are welcome!