MCP Doc Scraper

🚀 Document Scraper MCP Server

A Model Context Protocol (MCP) server that provides document scraping functionality. This server uses the conversion service of jina.ai to convert web-based documents into Markdown format.

🚀 Quick Start

This is a Model Context Protocol (MCP) server offering document scraping capabilities. It leverages jina.ai's conversion service to transform web - based documents into Markdown format.

✨ Features

• Scrape documents from any web page URL.
• Convert HTML documents to Markdown format.
• Save the converted documents to a specified output path.
• Integrate with Model Context Protocol (MCP).

📦 Installation

Install via Smithery

Automatically install the Doc Scraper for Claude Desktop through Smithery:

npx -y @smithery/cli install @askjohngeorge/mcp-doc-scraper --client claude

Manual Installation

1. Clone the repository:

git clone https://github.com/askjohngeorge/mcp-doc-scraper.git
cd mcp-doc-scraper

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -e .

💻 Usage Examples

Basic Usage

Run the server using Python:

python -m mcp_doc_scraper

Tool Description

The server provides one tool:

• Name: scrape_docs
• Description: Scrape documents from a URL and save them as Markdown files.
• Input Parameters:
  • url: The URL of the document to be scraped.
  • output_path: The path to save the Markdown file.

📚 Documentation

Project Structure

doc_scraper/
├── __init__.py
├── __main__.py
└── server.py

Dependencies

• aiohttp
• mcp
• pydantic

Development

1. Install development dependencies:

pip install -r requirements.txt

2. This server uses the Model Context Protocol. Please familiarize yourself with the MCP documentation.

📄 License

MIT License