Documentation Management System

🚀 DocMCP Documentation

DocMCP is a system designed for indexing the latest documentation of Large Language Models (LLMs). It can be integrated into AI IDEs through PostgreSQL and pgVector, aiming to provide efficient document storage, processing, and retrieval functions.

🚀 Quick Start

📦 Installation

Use the following command to install the project dependencies:

npm install

🏃‍♂️ Starting the Application

Run the following command to start the DocMCP system:

node src/main.ts

✨ Features

DocMCP offers the following core features:

1. Documentation Input

   • Users provide a URL via the add_documentation MCP tool.
   • The system creates a job record with a status of "Pending".
   • Jobs are assigned tags for categorization and filtering.

2. Web Crawling (CrawlerService)

   • The crawler adheres to the robots.txt rules.
   • It follows links up to a specified maximum depth.
   • Captures HTML content and metadata.
   • Creates document records associated with the parent job.

3. Document Processing (DocumentProcessorService)

   • Cleans HTML and converts it to structured Markdown.
   • Extracts metadata (package information, version, documentation type).
   • Establishes parent - child relationships between documents.
   • Updates the job progress.

4. Chunking and Embedding Generation (ChunkService)

   • Splits documents into semantic chunks to improve retrieval efficiency.
   • Generates vector embeddings using AWS Bedrock.
   • Stores the embeddings in PostgreSQL with the pgVector extension.
   • Retains chunk metadata and document references.

5. Job Completion (JobService)

   • Updates the job status to "Completed".
   • Calculates and stores document statistics.
   • Provides document query functionality.

6. Query and Retrieval

   • Users send queries via the query_documentation MCP tool.
   • The system converts the query into a vector embedding.
   • Performs a similarity search to find relevant chunks.
   • Returns formatted results, including source information.
   • Supports filtering by tags, status, and metadata.

📚 Documentation

Project Structure

docmcp/
├── prisma/                  # Database schema and migrations
│   └── schema.prisma        # Prisma model definitions and database configuration
├── src/
│   ├── config/              # Application configuration
│   │   └── database.ts      # Database connection settings
│   ├── generated/           # Generated code (Prisma client)
│   ├── services/            # Core service modules
│   │   ├── crawler.service.ts     # Website crawling functionality
│   │   ├── document.service.ts    # Document management
│   │   ├── document-processor.service.ts # Document processing and conversion
│   │   ├── job.service.ts         # Asynchronous job management
│   │   ├── chunk.service.ts       # Document chunking and vector operations
│   │   └── mcp-tools/       # MCP integration tools
│   │       ├── add-documentation.tool.ts    # Tool for adding new documentation
│   │       ├── get-job-status.tool.ts       # Tool for checking job status
│   │       ├── list-documentation.tool.ts   # Tool for listing available documentation
│   │       ├── query-documentation.tool.ts  # Tool for querying documentation
│   │       ├── sample.tool.ts               # Example tool implementation
│   │       └── index.ts                     # Tool registration and export
│   ├── types/               # TypeScript type definitions
│   │   └── mcp.ts           # MCP tool interface definitions
│   ├── utils/               # Utility functions
│   │   ├── logger.ts        # Logging utility
│   │   └── others.ts        # Other functions
│   └── main.ts              # Application entry point
├── package.json             # Dependency management file
└── README.md                # Project documentation

💻 Usage Examples

Basic Usage

Web Crawling Service (crawler.service.ts)

export class CrawlerService {
    async crawlUrl(url: string): Promise<Document> {
        // Implement crawler logic
    }
}

Document Processing Service (document-processor.service.ts)

export class DocumentProcessorService {
    async processDocument(document: Document): Promise<ProcessedDocument> {
        // Implement document processing logic
    }
}

🔚 Conclusion

DocMCP provides an efficient and flexible document management solution suitable for scenarios requiring large - scale document processing and retrieval. By combining web crawling, document processing, and vector embedding technologies, the system can meet the needs of various application scenarios.