Docs Fetch MCP

🚀 Document Fetching MCP Server

A Model Context Protocol (MCP) server for fetching web content with recursive exploration capabilities. This server enables Large Language Models (LLMs) to autonomously explore web pages and documents to learn about specific topics.

🚀 Quick Start

The Document Fetching MCP Server offers a simple yet powerful approach for retrieving and exploring web content. It allows:

• Fetching clean and readable content from any web page.
• Recursively exploring linked pages within a specified depth limit.
• Traversing links within the same domain to obtain comprehensive information.
• Intelligently filtering navigation links to focus on content-rich pages.

This tool is particularly useful when users want LLMs to learn about specific topics by exploring document or web page content.

✨ Features

• Content Extraction: Cleanly extract the main content from web pages, removing distractions such as navigation, ads, and irrelevant elements.
• Link Analysis: Identify and extract links on the page and evaluate their relevance.
• Recursive Exploration: Traverse relevant link content within the same domain, with a specified depth limit.
• Parallel Processing: Efficiently perform content fetching, supporting concurrent requests and error handling.
• Robust Error Handling: Gracefully handle network issues, timeouts, and incorrectly formatted pages.
• Dual-Strategy Approach: First use fast axios requests, and use puppeteer as a backup for more complex pages.
• Timeout Prevention: Implement global timeout handling to ensure reliable operation within the MCP time limit.
• Partial Results: Return the obtained content even if some pages fail to load completely.

💻 Usage Examples

Basic Usage

The server exposes a single MCP tool:

fetch_doc_content

Fetch web page content and recursively explore linked pages.

Parameters:

• url (string, required): The URL of the web page to fetch.
• depth (number, optional, default: 1): The maximum depth (1 - 5) for directory/link exploration.

Returns:

{
  "rootUrl": "https://example.com/docs",
  "explorationDepth": 2,
  "pagesExplored": 5,
  "content": [
    {
      "url": "https://example.com/docs",
      "title": "Documentation",
      "content": "Main page content...",
      "links": [
        {
          "url": "https://example.com/docs/topic1",
          "text": "Topic 1"
        },
        ...
      ]
    },
    ...
  ]
}

📦 Installation

1. Clone this repository:

git clone https://github.com/wolfyy970/docs-fetch-mcp.git
cd docs-fetch-mcp

2. Install dependencies:

npm install

3. Build the project:

npm run build

4. Configure your MCP settings in your Claude client:

{
  "mcpServers": {
    "docs-fetch": {
      "command": "node",
      "args": [
        "/path/to/docs-fetch-mcp/build/index.js"
      ],
      "env": {
        "MCP_TRANSPORT": "pipe"
      }
    }
  }
}

🔧 Technical Details

Dependencies

• @modelcontextprotocol/sdk: MCP server SDK
• puppeteer: Headless browser for web page interaction
• axios: HTTP client for sending requests

Development

To run the server in development mode:

npm run dev

📄 License

MIT