Webscraping Ai MCP Server

🚀 WebScraping.AI MCP Server Documentation

WebScraping.AI offers a server based on the Model Context Protocol (MCP) for integration with Large Language Models (LLMs). This server implements the Model Context Protocol and supports any MCP-enabled LLM platform.

🚀 Quick Start

📦 Installation

To use the WebScraping.AI server, follow these steps:

Clone the repository

git clone https://github.com/webscraping-ai/webscraping-ai-mcp-server.git
cd webscraping-ai-mcp-server

Install dependencies

npm install

⚙️ Configuration

Create and edit the .env file, adding the following environment variables:

• WEBSCRAPING_AI_API_KEY: Your WebScraping.AI API key
• Other optional environment variables such as proxy and log level

Here is an example of the .env file content:

WEBSCRAPING_AI_API_KEY=your-api-key
# Optional proxy settings
HTTP_PROXY=http://example.com:1080
HTTPS_PROXY=https://example.com:1080
NO_PROXY=localhost,127.0.0.1
# Log level options: debug, info, warning, error, critical
LOG_LEVEL=info

💻 Usage Examples

Basic Usage

Here is an example of using the webscraping-ai-mcp command-line tool:

npx webscraping-ai-mcp --url https://example.com --output json

Parameter Options

• --url: The target URL to scrape (required)
• --output: Output format, supporting text or json
• Other advanced parameters such as proxy settings and request headers

Tools List

Here are the available WebScraping.AI tools:

1. Web Page Scraping Tool

{
  "name": "webscraping_ai_fetch",
  "description": "Scrape web page content from the target URL.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": { "type": "string", "description": "The URL to scrape" },
      "output_format": { "type": "string", "enum": ["text", "json"], "default": "text" }
    }
  }
}

2. Website Analysis Tool

{
  "name": "webscraping_ai_analyze",
  "description": "Analyze the technical details of the target website.",
  "parameters": {
    "type": "object",
    "properties": {
      "url": { "type": "string", "description": "The URL to analyze" }
    }
  }
}

3. Account Information Tool

{
  "name": "webscraping_ai_account",
  "description": "Get WebScraping.AI account information.",
  "parameters": {
    "type": "object",
    "properties": {}
  }
}

🔧 Technical Details

Error Handling

The server provides powerful error handling capabilities, including:

• Automatic retry mechanism
• Network connection recovery
• Request rate limit handling
• Detailed error logs

Example Error Response

{
  "error": {
    "code": "403",
    "message": "Access denied",
    "details": "The request exceeded the API quota"
  }
}

Integration with LLM

The server supports integration with LLMs through the following steps:

Step 1: Install dependencies

npm install web-scraping-ai mcp-service

Step 2: Create a service

const { MCPService } = require('mcp-service');
const service = new MCPService({
  apiKey: 'your-api-key',
  provider: 'openai'
});

🛠️ Development

Start the development server

npm run dev

Test commands

npm test

🤝 Contribution Guidelines

Submit code

1. Create a feature branch
2. Commit changes and push
3. Create a pull request

📄 License

This project follows the MIT License, with the following details:

• Free use, copying, modification, and distribution of the code are permitted.
• A copyright notice must be included in the distributed work.