Web Search Server

🚀 Web Search MCP Server

A powerful MCP server that offers Google search functionality, web content viewing, and advanced anti-bot detection circumvention capabilities.

🚀 Quick Start

The Web Search MCP Server is a robust tool designed to perform complex tasks, such as Google searches, web content parsing, and other internet interaction-related operations. It is specifically designed for integration into the Model Context Protocol (MCP) environment, providing a scalable and secure way to handle online data.

✨ Features

• Custom Google Search: Supports custom search domains and excludes specific domain names.
• Web Content Viewing: Extracts content from specified URLs and supports multiple formatting options.
• Rate Limiting and Resource Management: Optimizes performance through a browser instance pool and automatic cleanup of idle instances.
• Anti-bot Detection Circumvention: Integrates advanced techniques to bypass common anti-bot detection mechanisms.

📦 Installation

Prerequisites

1. Node.js and npm need to be installed on the system.
2. Ensure that the typescript and @types/node packages are installed:

   npm install -g typescript @types/node --save-dev
   

Steps

# Clone the repository
git clone https://github.com/your-repository.git
cd web-search-mcp-server

# Install dependencies
npm install

📚 Documentation

Configuration

1. Create a Configuration File

Create a config.json file in the project root directory and add the following content:

{
  "googleApiKey": "YOUR_GOOGLE_API_KEY",
  "proxySettings": {
    "http": "HTTP_PROXY_URL",
    "https": "HTTPS_PROXY_URL"
  },
  "browserOptions": {
    ".userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.128 Safari/537.36",
    "disableImages": false
  }
}

2. Set Environment Variables

Set the following environment variables in the terminal:

export MCP_SERVER_ENV=development

Cookie Settings

To use the cookie feature, follow these steps:

1. Download and install the Chrome browser.
2. Install the uBlock Origin extension.
3. Export your browser profile:

   chrome://settings/resetBrowser
   

4. Set the exported profile path as an environment variable:

   export BROWSER_PROFILE=~/path/to/your/exported/profile
   

Logging

• Real-time Log Viewing:
  • On MacOS/Linux:

    tail -n 20 -f ~/Library/Logs/MCP/mcp*.log
    

  • On Windows:

    type %APPDATA%\Claude\Logs\mcp*.log
    

Anti-bot Detection

Core Features

• Browser Pool: Optimizes performance by reusing browser instances.
• Request Frequency Control: Prevents triggering anti-bot mechanisms due to excessive requests.
• Circumvention Techniques:
  • Dynamic timeout adjustment
  • Random user agents
  • Simulation of human behavior patterns

Common Issues

1. Server not displaying:

• Check the syntax and path of the configuration file.

2. Tool invocation failure:

• Check the server logs and restart Claude Desktop.

3. Path issues:

• Ensure that absolute paths are used.

Using Claude Desktop

Configuration Steps

1. Open the config.json file.
2. Add the following to the tool list:

   {
     "name": "search",
     "description": "Search the internet and return results"
   },
   {
     "name": "view_url",
     "description": "View the content of a specified URL and return text"
   }
   

Confirm Integration

1. Check the tool icon:

• In Claude Desktop, confirm the appearance of the hammer icon (refer to the picture).

Available Tools

1. Search Tool

{
  name: "search",
  params: {
    query: string;
    trustedDomains?: string[];
    excludedDomains?: string[];
    resultCount?: number;
    safeSearch?: boolean;
    dateRestrict?: string;
  }
}

2. View URL Tool

{
  name: "view_url",
  params: {
    url: string;
    includeImages?: boolean;
    includeVideos?: boolean;
    preserveLinks?: boolean;
    formatCode?: boolean;
  }
}

Troubleshooting

• Common Errors:
  • Network connection issues
  • Anti-bot detection triggered
  • Configuration file syntax errors
• Solutions:
  • Check the network connection
  • Slow down the request frequency
  • Ensure the configuration file is correct

Development

Project Structure

web-search-mcp-server/
├── src/                 # Core source code directory
│   ├── browser.ts      # Browser management logic
│   ├── search.ts       # Search function implementation
│   └── view.ts         # View URL function implementation
├── config.json          # Project configuration file
└── package.json         # Dependency management file

🤝 Contribution

We welcome contributions from the community. Please fork the repository and submit a pull request.

📄 License

MIT License

📞 Contact

• Email: contact@example.com
• GitHub: https://github.com/your-repository

By following the above steps, you can easily set up and run the Web Search MCP Server and integrate it into your project.