Mcpdocsearch

This project provides a set of toolkits for crawling website content and generating Markdown documents. At the same time, it realizes the semantic search function of documents through the MCP server and supports integration with tools such as Cursor.

Developer tools Search tools #Document crawling #Semantic search #MCP service #Intelligent Q&A .Python

rating : 2.5 points

downloads : 7.4K

update time : 2025-04-29

Open Site

What is the Document Crawling and MCP Search Server?

This is an intelligent toolkit that can automatically crawl website document content, convert it into a structured format, and help you quickly find the information you need through semantic search technology. It is particularly suitable for retrieving technical content such as development documents and API references.

How to use this service?

Simply provide the URL of the target document website, and the tool will automatically crawl the content and establish a search index. Then you can use natural language queries to find relevant content, just as simple as using a smart assistant.

Applicable scenarios

It is particularly suitable for developers, technical support teams, and technical writers who need to frequently consult large technical documents. It can significantly improve the efficiency of finding information in complex documents.

Main features

Intelligent web crawling

Automatically traverse the website structure and crawl document content. The crawling depth and scope can be configured.

Intelligent content processing

Automatically clean up irrelevant content (navigation bars, footers, etc.) and retain the core document content.

Semantic search

Use AI technology to understand the query intention and find the most relevant content fragments, rather than simple keyword matching.

Cursor integration

Seamlessly integrate into the Cursor IDE, allowing you to directly query documents during development.

Intelligent cache system

Automatically cache the processing results for faster loading in subsequent use.

Advantages

Save time on manually searching for documents

Understand natural language queries without relying on precise keywords

Customizable crawling scope and depth

Automatically keep documents up to date

Support in - depth retrieval of complex technical documents

Limitations

It takes a long time to process a large document set for the first time

Limited support for highly dynamic pages rendered by JavaScript

Requires reasonable configuration of crawling parameters to achieve the best results

Does not support image content recognition for now

How to use

Installation preparation

Ensure that Python and the uv tool are installed, and clone the project repository.

Crawl documents

Run the crawling command and specify the URL of the target document website.

Configure Cursor integration

Create a.cursor/mcp.json configuration file in the project root directory.

Start searching

Use the @doc - query - server command in Cursor to query document content.

Usage examples

Crawl API documentation

Only crawl the API reference part of the website

Exclude specific content

Crawl documents but exclude the blog and example parts

Process SPA websites

Crawl single - page application documents rendered by JavaScript

Frequently Asked Questions

Why is the first startup of the server slow?

How to update the crawled documents?

What types of websites are supported?

What is the appropriate crawling depth setting?

Why is some page content missing?

Related resources

Project code repository

Source code and latest updates

Cursor IDE official website

Learn how to integrate with Cursor

Model Context Protocol

Official documentation of the MCP protocol

Install the uv tool

Project dependency management tool

🚀 MCP Document Query Server Project

This project aims to build a document query server based on the MCP protocol through web crawling, data cleaning, and vector embedding generation. Users can access this server via configuration tools (such as Cursor) and perform tasks like document search.

🚀 Quick Start

📦 Installation

Environment Setup

Create a project environment using uv:

uv init --python 3.8 mcp-doc-search
cd mcp-doc-search

Activate the virtual environment and install dependencies:
```
uv activate
pip install -r requirements.txt
```

Configuration Instructions

Project Path: Replace /path/to/your/MCPDocSearch with the actual absolute path of the project.
Port Configuration: By default, --port 12345 is used. You can modify it as needed.

💻 Usage Examples

Web Crawling

Run the web crawler tool to obtain document content:

uv run python -m crawler_cli.cli --help

Starting the MCP Server

Start the MCP server and load document data:

uv run python -m mcp_server.main

📚 Documentation

Project Architecture

crawler_cli: Users provide the starting URL and parameters through the command - line tool.
Web Crawling (crawl4ai): Use the crawl4ai library to crawl web page content according to rules.
Data Cleaning: An optional step. Use BeautifulSoup to clean navigation bars and links in HTML.
Markdown Generation (crawl4ai): Convert the cleaned HTML to Markdown format.
Document Storage: Store the generated Markdown files in the ./storage/ directory.
Starting the MCP Server (mcp_server.data_loader):
- Load data from cache files or directly read .md files.
- Process Markdown documents in chunks by title and generate text embeddings.
Tool Exposure (mcp_server.mcp_tools): Expose tools such as document query through fastmcp.
Query Execution (Cursor): Call server tools through the MCP client to implement the document search function.

Dependencies

crawl4ai: Core web - crawling functionality.
fastmcp: Server implementation of the MCP protocol.
sentence - transformers: Text embedding generation.
torch: Used for sentence - transformers.
typer: Build command - line tools.
uv: Project and environment management.
beautifulsoup4: HTML parsing (through crawl4ai).
rich: Terminal output enhancement.

📄 License

This project is licensed under the MIT License. For specific terms, please refer to the LICENSE file.

🔧 Technical Details

Security Precautions

⚠️ Important Note

The project uses Python's pickle module for data caching. It is recommended to ensure that the ./storage/ directory is writable only by trusted users and processes.