WEB SCRAPING MCP

🚀 Network Crawler Project Introduction

This project is a network crawler system based on the MCP protocol, supporting deployment via Docker containers or local operation. The following is the complete documentation for the project.

🚀 Quick Start

✨ Features

• Based on the MCP protocol.
• Supports deployment via Docker containers or local operation.

📦 Installation

Docker Deployment

1. Install Docker Ensure that Docker is installed and running on your system. For Windows or Mac users, please refer to the Docker official website for installation.
2. Build the Image In the project root directory, execute the following command to build the Docker image:

   docker build -t your-image-name .
   

3. Run the Container Use the following command to start the Docker container:

   docker run --env-file .env -p 8002:8002 your-image-name
   

   • --env-file .env: Load the environment variable file.
   • -p 8002:8002: Map port 8002 of the host machine to port 8002 inside the container.
4. Access the Service After the container is started, you can access the service via http://localhost:8002/sse. Configure the MCP client (such as the LangChain agent) to connect to this address.

Local Operation

1. Install Python Ensure that Python 3.9 or a higher version is installed. It is recommended to use Python 3.10 or above.
2. Create a Virtual Environment In the project directory, execute the following command to create and activate the virtual environment:

   python -m venv venv
   source venv/bin/activate  # Linux/macOS
   # venv\Scripts\activate  # Windows
   

3. Install Dependencies Install the Python packages required for the project:

   pip install -r requirements.txt
   

4. Configure Environment Variables Create a file named .env and add the following content (fill in according to the actual API key):

   GOOGLE_API_KEY=your_api_key_here
   

5. Run the Service Execute the following command to start the service:

   python your_server_script_name.py
   

🔧 Technical Details

Environment Variables

The environment variables used in the project are as follows (usually loaded from the .env file):

💻 Usage Examples

Basic Usage

# Use the agent CLI to perform the following operations:

You: scrape_url https://example.com
Agent: Thinking...
[Agent calls the scrape_url tool]
Agent: [Markdown content of example.com]

---
You: Extract text containing "ethical considerations" from https://en.wikipedia.org/wiki/Web_scraping
Agent: Thinking...
[Agent calls the extract_text_by_query tool]
Agent: X matches were found on the page. Here are the relevant paragraphs:
Match 1:
... text snippet ...
---
Match 2:
... text snippet ...

---
You: Use smart_extract to analyze https://blog.google/technology/ai/google-gemini-ai/ and obtain the main points of the Gemini model
Agent: Thinking...
[Agent calls the smart_extract tool]
Agent: Gemini is an advanced AI model developed by Google, with multi - language and cross - domain understanding capabilities. It can access the latest information through the Internet and support context - aware conversations.

📚 Documentation

Project Files

• Dockerfile: Used to build the Docker image.
• requirements.txt: Python dependencies for the project.
• .env: Environment variable configuration file.

📄 License

No license information is provided in the original document.

Summary

This project provides a network crawler system based on the MCP protocol, supporting operation via Docker or locally. With simple configuration and environment variable settings, you can quickly deploy and use the relevant functions.