MCP Windows Website Downloader

🚀 MCP Website Downloader

A simple MCP server for downloading documentation websites and preparing them for RAG indexing.

✨ Features

• Download a complete documentation site, or at least a large part of it.
• Maintain the link structure and navigation. Well, not yet, lol.
• Download and organize resources (CSS, JS, images), but it's not AI-friendly. All content may need some form of parsing or vectorization into a database.
• Create a clean index for the RAG system. Currently, it seems to create an index in each folder without even checking what it looks like, lol.
• A simple single-purpose MCP interface. Yes.

📦 Installation

Fork and download the repository, then enter the repository directory:

uv venv
./venv/Scripts/activate
pip install -e .

Add the following content to your claude_desktop_config.json (according to your own path):

   "mcp-windows-website-downloader": {
     "command": "uv",
     "args": [
       "--directory",
       "F:/GithubRepos/mcp-windows-website-downloader",
       "run",
       "mcp-windows-website-downloader",
       "--library",
       "F:/GithubRepos/mcp-windows-website-downloader/website_library"
     ]
   },

💻 Usage Examples

Basic Usage

1. Start the server:

python -m mcp_windows_website_downloader.server --library docs_library

2. Call through Claude Desktop or other MCP clients:

result = await server.call_tool("download", {
    "url": "https://docs.example.com"
})

📚 Documentation

Output Structure

docs_library/
  domain_name/
    index.html
    about.html
    docs/
      getting-started.html
      ...
    assets/
      css/
      js/
      images/
      fonts/
    rag_index.json

Development

The server follows the standard MCP architecture:

src/
  mcp_windows_website_downloader/
    __init__.py
    server.py    # MCP server implementation
    core.py      # Core website download function
    utils.py     # Auxiliary tools

Components

• server.py: The main MCP server implementation, handling tool registration and requests.
• core.py: The core website download function, responsible for resource processing.
• utils.py: Auxiliary tools for file processing and URL parsing.

Design Principles

1. Single Responsibility
   • Each module has only one clear responsibility.
   • The server handles the MCP interface.
   • The core handles downloading.
   • The tools handle general operations.
2. Clear Structure
   • Maintain the original website structure.
   • Organize resources by type.
   • Create a clear index for the RAG system.
3. Robust Operation
   • Reasonable error handling.
   • Download depth limit.
   • Resource download verification.
   • Clear URL/path handling.

RAG Index

The rag_index.json file contains:

{
  "url": "https://docs.example.com",
  "domain": "docs.example.com", 
  "pages": 42,
  "path": "/path/to/site"
}

🤝 Contribution

1. Fork the repository.
2. Create a feature branch.
3. Make your modifications.
4. Submit a pull request.

📄 License

MIT License - See the LICENSE file for details.

🔧 Technical Details

Error Handling

The server handles common issues:

• Invalid URLs
• Network errors
• Resource download failures
• HTML format errors
• Download depth limits

Example error response:

{
  "error": {
    "code": 500,
    "message": "Internal Server Error"
  }
}

Example Output

Example successful response:

{
  "status": {
    "code": 200,
    "message": "Download completed successfully"
  },
  "data": {
    "downloadedFiles": ["index.html", "about.html", "docs/getting-started.html"],
    "totalFiles": 3,
    "failedDownloads": []
  }
}