MCP Server For Document Processing

🚀 MCP Server Operation Guide

MCP is a multi - document processing system designed for text analysis in the scientific, technological, and financial fields. It includes an efficient pipeline and a Web server, supporting various language processing functions.

🚀 Quick Start

📦 Installation

1. Clone the repository

   git clone https://github.com/donphi/MCP.git
   cd MCP
   

2. Install dependencies

   • For the pipeline:

     pip install -r requirements.pipeline.txt
     

   • For the server:

     pip install -r requirements.server.txt
     

3. Configure environment variables Set the following variables in the .env file:

   CHUNK_SIZE=500
   OVERLAP=100
   MODEL_NAME=sentence-transformers/all-mpnet-base-v2
   

   If you want to use a free local model, comment out MODEL_NAME.

4. Start the pipeline

   python src/pipeline.py
   

5. Start the server

   python src/server.py
   

✨ Features

Pipeline Components

Document Processing Workflow

1. Document Loading: Supports multiple formats, including PDF, DOCX, and text files.
2. Type Detection: Uses spaCy for language model training to identify document types.
3. Chunking Strategies:
   • Scientific Papers: Split by sections while retaining references.
   • Financial Reports: Keep tables and numerical parts.
   • Technical Documents: Preserve code blocks and examples.
   • General Text: Use spaCy's semantic boundary recognition.

Pipeline Tools

• enhanced_chunking.py: Provides a more intelligent chunking method.
• inspect_chunks.py: Analyzes the existing chunking structure.
• visualize_chunks.py: Visualizes the chunking results.

Server Functions

Core Services

1. Document Storage:

   • Uses ChromaDB to store the vector database.
   • Supports integration with distributed file systems.

2. Query Processing:

   • Multi - modal retrieval (text + image).
   • Supports natural language queries.
   • Provides relevance scoring and context analysis.

Other Features

• Document Management: Includes version control and access permission settings.
• Statistical Analysis: Provides chunking efficiency reports.
• Logging System: Records all operations and error information.

🔧 Technical Details

Troubleshooting

• Docker not found: Ensure Docker is installed and running. You can check with docker --version.
• "Invalid reference format" error:
  • Reason 1: The build step was not executed. Run docker-compose build server first.
  • Reason 2: A combination of Docker and Docker Compose is used.
• API key issues: You can use a free local model without a key.
• Missing sentence - transformers package: The system will automatically install the required packages.
• Chroma database not found: Ensure the pipeline has been run to process documents.
• Connection issues: Check the MCP configuration for correctness.
• Low chunking efficiency: Adjust the CHUNK_SIZE and OVERLAP parameters.

Optimization Suggestions

1. Performance Tuning:

   • Adjust the parallel processing threads according to the number of CPU cores.
   • Use SSDs to improve storage speed.

2. Model Selection:

   • Choose an appropriate embedding model based on your needs.
   • For large - scale data, use distributed training.

3. Scalability:

   • Deploy to a Kubernetes cluster.
   • Configure an automatic scaling strategy.

Security Precautions

• Access Control: Use HTTPS and authentication.
• Log Monitoring: Implement real - time log analysis.
• Regular Backups: Set up automated backup tasks.

📚 Documentation

Frequently Asked Questions

1. How to handle large files?
   • Upload in chunks and process in parallel.
2. Which languages are supported?
   • English, Chinese, and some other European languages.
3. Does it support incremental updates?
   • Yes, it provides version control functionality.

Contact Information

• Email: support@MCP.com
• GitHub: https://github.com/donphi/MCP

This guide details the installation, configuration, and usage of the MCP server, and provides troubleshooting and optimization suggestions.