Document Crawler & Search

🚀 MCPDocSearch

A knowledge base system built using crawl4ai and fastmcp protocols, enabling information retrieval from documents via natural language queries.

🚀 Quick Start

This README will guide you through the installation, usage, and configuration of the MCPDocSearch project.

✨ Features

• Construct a knowledge base system with crawl4ai and fastmcp protocols.
• Retrieve information from documents using natural language queries.
• Integrate with Cursor/Claude desktop versions.

📦 Installation

Project Overview

This is a knowledge base system built using the crawl4ai and fastmcp protocols. The system allows users to retrieve information from documents through natural language queries.

Installation Steps

1. Clone the repository:

   git clone git@github.com:yourusername/MCPDocSearch.git
   cd MCPDocSearch
   

2. Create a virtual environment and install dependencies:

   uv env create --python 3.8+
   pip install -r requirements.txt
   

📚 Documentation

Document Structure

The project consists of the following key components:

1. crawler_cli: Responsible for web crawling and document generation.
2. mcp_server: Implements the MCP protocol and provides a knowledge base query interface.

Core Process

1. Run the crawler:

   • Use crawl4ai to start crawling from the specified URL.
   • Fetch page content according to rules (e.g., depth, link patterns).

2. Content cleaning (optional):

   • Use BeautifulSoup to clean HTML and remove non - essential elements such as navigation bars.

3. Document generation:

   • Convert the cleaned HTML to Markdown format.
   • Save it to the ./storage/ directory.

4. Start the MCP server:

   uv run python -m mcp_server.main
   

Configuration Examples

Integration with Cursor/Claude Desktop Version

Create a .cursor/mcp.json file in the project root directory with the following content:

{
  "mcpServers": {
    "doc-query-server": {
      "command": "uv",
      "args": [
        "--directory",
        "/your/MCPDocSearch/absolute/path",
        "run",
        "python",
        "-m",
        "mcp_server.main"
      ],
      "env": {}
    }
  }
}

Security Tips

• Pickle Cache: The project uses the pickle module to cache and process data. To ensure security, make sure the ./storage/ directory is only accessed by trusted users or processes.

Dependencies

Key libraries include:

• crawl4ai: Core web crawling functionality
• fastmcp: MCP server implementation
• sentence-transformers: Text embedding generation
• torch: Dependency for sentence-transformers
• typer: Building command - line tools
• uv: Project and environment management
• beautifulsoup4 (used through crawl4ai)
• rich: Enhanced terminal output

Architecture

The system follows the following process:

1. Crawler CLI: Users run the tool, specifying the starting URL and related options.
2. Web Crawling: Use crawl4ai to obtain web page content and fetch links according to rules.
3. Content Cleaning (optional): Use BeautifulSoup to clean HTML.
4. Document Generation: Convert the processed content to Markdown format and save it.
5. MCP Server Startup: Run the mcp_server.main module to provide knowledge base query services.

Security Precautions

• Ensure that the cache directory ./storage/ is only accessed by authorized users.
• Regularly clean old documents to reduce potential security risks.