Markdown Web Crawl

🚀 MD Website Crawler Project (Based on MCP Protocol)

A Python-based web crawler using the MCP protocol (https://modelcontextprotocol.io/introduction) to extract and save website content.

🚀 Quick Start

This is a Python-based web crawler that uses the MCP protocol to extract and save website content.

✨ Features

• Extract website content and save it in Markdown file format.
• Map website structure and links.
• Process multiple URLs in batches.
• Configurable output directory.

📦 Installation

1. Clone the repository:

git clone https://github.com/yourusername/webcrawler.git
cd webcrawler

2. Install dependencies:

pip install -r requirements.txt

3. Optional: Configure environment variables:

export OUTPUT_PATH=./output  # Set your preferred output directory

📚 Documentation

Output Results

The crawled content will be saved in Markdown format in the specified output directory.

Configuration Options

Configure the server through environment variables:

Using Claude Settings

Install the server via FastMCP: fastmcp install server.py

Or run it directly with custom settings:

{
    "Crawl Server": {
        "command": "fastmcp",
        "args": [
            "run",
            "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
        ],
        "env": {
            "OUTPUT_PATH": "/Users/user/Webcrawl"
        }
    }
}

Development Guide

Real-time Development

fastmcp dev server.py --with-editable .

Debugging

It is recommended to use the tool at https://modelcontextprotocol.io/docs/tools/inspector for debugging.

💻 Usage Examples

Example 1: Extract and Save Content

mcp call extract_content --url "https://example.com" --output_path "example.md"

Example 2: Create Content Index

mcp call scan_linked_content --url "https://example.com" | \
  mcp call create_index --content_map - --output_path "index.md"

🔧 Contribution Guide

1. Fork the repository.
2. Create a feature branch (git checkout -b feature/AmazingFeature).
3. Commit your changes (git commit -m 'Add some AmazingFeature').
4. Push to the branch (git push origin feature/AmazingFeature).
5. Submit a Pull Request.

📄 License

This project is open-source under the MIT license. For more details, please refer to the LICENSE file.

Dependencies

• Python 3.7+
• FastMCP (uv pip install fastmcp)
• Other dependencies listed in requirements.txt