URL Fetch

🚀 URL Link Fetching MCP

A clean implementation of Model Context Protocol (MCP) that enables Claude or any LLM to retrieve content from URLs.

✨ Features

• Retrieve content from any URL.
• Support multiple content types (HTML, JSON, text, images).
• Control request parameters (headers, timeout).
• Provide clear error handling.
• Support interactive CLI operations.

📁 Directory Structure

Project Root/
├── mcp_urlfetch/                # Core code module
│   ├── __init__.py             # Package initialization file
│   ├── client.py               # Client API implementation
│   ├── server.py               # MCP server implementation
│   └── tools.py                # Definition of utility functions
├── examples/                   # Example scripts
│   ├── quick_test.py           # Quick test script
│   ├── simple_usage.py         # Simple usage example
│   └── interactive_client.py   # Interactive CLI client
└── tests/                      # Test cases
    ├── direct_test.py          # Direct test for URL fetching
    └── mcp_server_test.py      # Test for MCP server functionality

📦 Installation

Install Dependencies

pip install "requests>=2.31.0" "python-multipart>=0.3.5"

💻 Usage Examples

🚀 Quick Start

Start the MCP Server

python mcp_urlfetch/server.py --port 8000

Call Utility Functions

Use in Python code:

from mcp_urlfetch.client import AsyncMCPClient

async def main():
    async with AsyncMCPClient('localhost', 8000) as client:
        # Use the fetch_url tool
        result = await client.call_tool("fetch_url", {
            "url": "https://example.com"
        })
        
        # Use the fetch_json tool
        result = await client.call_tool("fetch_json", {
            "url": "https://api.example.com/data",
            "headers": {"Authorization": "Bearer token"}
        })
        
        # Use the fetch_image tool
        result = await client.call_tool("fetch_image", {
            "url": "https://example.com/image.jpg"
        })

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

🔧 Core Implementation

Define Utility Functions

Defined in mcp_urlfetch/tools.py:

from typing import Dict, Optional
from pydantic import Field
from mcp_core.context import Context
from mcp_core.tool import Tool
from mcp_core.utils import async_func_to_coroutine

@Tool(
    name="fetch_url",
    description="Fetch content from a URL and return it as text."
)
async def fetch_url(url: AnyUrl, headers: Optional[Dict[str, str]] = None, timeout: int = 10, ctx: Context = None):
    # Implementation details...

@Tool(
    name="fetch_image",
    description="Fetch an image from a URL and return it as an image."
)
async def fetch_image(url: AnyUrl, timeout: int = 10, ctx: Context = None):
    # Implementation details...

@Tool(
    name="fetch_json", 
    description="Fetch JSON from a URL, parse it, and return it formatted."
)
async def fetch_json(url: AnyUrl, headers: Optional[Dict[str, str]] = None, timeout: int = 10, ctx: Context = None):
    # Implementation details...

🛠️ Tool Functions

fetch_url

Fetch content from the specified URL and return it as text.

fetch_image

Fetch an image from the specified URL and return the image data.

fetch_json

Fetch JSON data from the specified JSON interface, parse it, and return a formatted string.

📋 Example Usage

Quick Test Script (examples/quick_test.py)

from mcp_urlfetch.client import AsyncMCPClient

async def main():
    async with AsyncMCPClient('localhost', 8000) as client:
        # Test the fetch_url tool
        result = await client.call_tool("fetch_url", {"url": "https://example.com"})
        print(f"URL content: {result}")
        
        # Test the fetch_json tool
        result = await client.call_tool("fetch_json", {
            "url": "https://api.example.com/data",
            "headers": {"Authorization": "Bearer token"}
        })
        print(f"JSON data: {result}")
        
        # Test the fetch_image tool
        result = await client.call_tool("fetch_image", {"url": "https://example.com/image.jpg"})
        print(f"Image data fetched, size: {len(result)} bytes")

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

🧪 Testing

Run Test Cases

python -m pytest tests/

📄 License

This project is licensed under the MIT License.

🤝 Contribution Guidelines

Feel free to fork this repository and submit a Pull Request!

📚 Documentation

• MCP Standard Protocol
• Python Requests Library
• Asynchronous Programming Guide

Project Version: 1.0.0
Last Document Update: March 1, 2024