Fetch MCP Server Y8a

🚀 Fetch MCP Server

A Model Context Protocol (MCP) server that offers web content scraping capabilities, leveraging browser automation, OCR technology, and various extraction methods. This server enables Large Language Models (LLMs) to retrieve and process content from internet pages, even those requiring JavaScript rendering or using anti-scraping techniques.

🚀 Quick Start

A Model Context Protocol (MCP) server provides web content scraping capabilities, utilizing browser automation, OCR technology, and various extraction methods. This server allows Large Language Models (LLMs) to retrieve and process content from internet pages, even if these pages require JavaScript rendering or use anti-scraping techniques.

✨ Features

Available Tools

• fetch - Scrapes URLs from the internet using browser automation and multi-method extraction (including OCR).
  • url (string, required): The URL to scrape.
  • raw (boolean, optional): Retrieve the actual HTML content instead of the simplified version (default: false).

The server uses multiple methods to extract content:

1. Browser automation with undetected-chromedriver.
2. OCR with pytesseract and layout detection.
3. HTML extraction with requests/BeautifulSoup.
4. Document parsing (PDF, DOCX, PPTX).
5. Raw Markdown conversion method.

The server uses a sophisticated scoring system to select the best result, considering the following factors:

1. Base Content Score (max 50 points)
   • Points are awarded based on content length (1 point per 100 characters, max 50 points).
   • Deductions are made for extremely short content (<100 characters).
2. Structure Bonus (max 20 points)
   • Points are awarded for well-structured content.
   • More paragraphs indicate better organization.
3. Quality Penalty
   • Error messages are detected and penalized.
   • Content containing error indicators is downgraded.
   • Content structure and readability are verified.

The scoring system ensures that the most reliable and high-quality content is selected, regardless of the extraction method used. Debug logs are provided to track scoring decisions.

Tips

• fetch
  • Scrape a URL and extract its content in Markdown format using browser automation.

Parameters:

• url (string, required): The URL to scrape.

📦 Installation

Using Docker

To install and run mcp-server-fetch using Docker, follow these steps:

1. Build the Docker image:

   docker build -t mcp-server-fetch .
   

2. Run the Docker container:

   docker run --rm -i mcp-server-fetch
   

📚 Documentation

Configuration

Configure Roo Code or Claude App

Add the following content to your Claude settings:

{
  "mcpServers": {
    "fetch": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "mcp-server-fetch"
      ],
      "disabled": false,
      "alwaysAllow": []
    }
  }
}

Custom - User Agent

By default, the server will use the following user agents depending on whether the request is from a model (via a tool) or initiated by a user (via a prompt):

ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)

or

ModelContextProtocol/1.0 (User-Specified; +https://github.com/modelcontextprotocol/servers)

You can customize the user agent string by adding the parameter --user-agent to the args list.

Browser Automation Settings

If you need to configure browser automation, refer to the following example:

{
  "mcpServers": {
    "fetch": {
      "command": "docker",
      "args": [
        "run",
        "--rm",
        "-i",
        "mcp-server-fetch"
      ],
      "customHeaders": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36"
      },
      "disableCache": true,
      "screenshots": {
        "enabled": true,
        "path": "/screenshots"
      }
    }
  }
}

Advanced Configuration

You can use the following advanced options:

• proxy: Set a proxy.
• timeout: Set the request timeout.
• maxRedirects: Set the maximum number of redirects.
• retry: Set the number of retries.
• delay: Set the scraping delay.

👥 Contribution

We welcome contributions from the community! Please refer to the Contribution Guidelines.

📄 License

This project is licensed under the MIT License:

MIT License

Copyright (c) 2023 Your Name

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

📞 Contact

• Email: contact@example.com
• GitHub: Your GitHub Link

Thank you for using the Fetch MCP Server! If you have any questions or suggestions, please feel free to contact us.