Kb MCP Server

🚀 Embedded Server

The MCP (Magic Content Processor) server offers a range of powerful features through the txtai toolkit, including knowledge base construction, content retrieval, and a question-answering system. Below are its core features and configuration guides.

🚀 Quick Start

📦 Install Dependencies

pip install txtai faiss-cpu

📚 Initialize the Knowledge Base

from txtai import AI
import os

# Initialize the AI model, using the local device by default
ai = AI()
os.environ["TOKENIZERS_PARALLELISM"] = "false"

# Load or create the knowledge base (adjust parameters according to specific needs)
knowledge_base = ai.build(
    path="/root/.txtai/embeddings",  # Knowledge base storage path
    content_path="sqlite:///~/.txtai/content.db",  # Content storage database path
    embeddings={
        "path": "sentence-transformers/nli-mpnet-base-v2",
        "backend": "faiss",
        "gpu": False,
        "batch": 32,
        "normalize": True
    },
    scoring="hybrid",  # Hybrid search strategy
    hybridalpha=0.75,  # Balance coefficient for hybrid search
    graph={
        "backend": "sqlite",
        "path": "~/.txtai/graph.db",
        "similarity": 0.75,
        "limit": 10
    },
    extractor="distilbert-base-cased-distilled-squad"  # Extraction model
)

📋 Knowledge Base Construction Configuration

📄 YAML Configuration Example

# Knowledge base storage path and persistence settings
path: ~/.txtai/embeddings
writable: true

# Content storage database configuration
content:
  path: sqlite:///~/.txtai/content.db

# Embedding model parameters
embeddings:
  # Model selection and device configuration
  path: sentence-transformers/nli-mpnet-base-v2
  backend: faiss
  gpu: false  # Adjust whether to use GPU acceleration according to hardware conditions
  batch: 32    # Batch size, affecting performance and memory usage
  normalize: true  # Whether to normalize the embedding vectors

  # Search strategy configuration
  scoring: hybrid  # Supports hybrid (text + semantic) search mode
  hybridalpha: 0.75  # Hybrid weight, the larger the value, the more dependent on semantic similarity

# Management process parameters
pipeline:
  workers: 2   # Number of concurrent processing threads
  queue: 100   # Request queue size
  timeout: 300  # Processing timeout (seconds)

# Extraction model configuration
extractor:
  path: distilbert-base-cased-distilled-squad
  maxlength: 512  # Maximum context length limit
  minscore: 0.3   # Minimum matching score threshold

# Knowledge graph construction parameters
graph:
  backend: sqlite  # Use SQLite to store graph data
  path: ~/.txtai/graph.db
  similarity: 0.75  # Similarity threshold for graph node connections
  limit: 10   # Maximum number of connections per node

🛠️ Knowledge Base Construction Process

Step 1: Install Necessary Packages

pip install txtai faiss-cpu transformers

Step 2: Select an Appropriate Configuration File

Based on specific needs, select or customize a YAML configuration from the following examples:

• memory.yml: Memory mode (suitable for quick testing)
• sqlite-faiss.yml: Use SQLite and FAISS for persistent storage
• postgres-pgvector.yml: Integrate PostgreSQL and the pg_vector extension

Step 3: Execute the Construction Command

txtai build --config config.yml

🏗️ System Architecture Overview

The MCP server adopts a modular design, and its main components include:

1. Knowledge Base Manager: Responsible for content storage, indexing, and retrieval.
2. Embedding Model: Used to convert text into vector representations, supporting multiple pre-trained models.
3. Question-Answering System: Provides intelligent answers based on embedding technology and a rule engine.
4. Extension Mechanism: Allows new processing modules to be added in the form of plugins.

⚙️ System Performance Tuning

Parameter Adjustment Suggestions

• Memory Usage: Adjust the batch size and the number of workers according to server resources to balance memory usage and processing efficiency.
• Search Strategy: Select an appropriate scoring method based on specific needs and optimize the hybridalpha value to achieve the best retrieval results.
• Hardware Acceleration: If GPU resources are available, it is recommended to enable the gpu parameter to obtain faster embedding calculation speeds.

Log Monitoring

Integrate a logging system (such as ELK) to monitor the server's running status and performance metrics in real-time, and promptly identify and solve problems.

🚧 Extension and Maintenance

Plugin Development

Developers can define interfaces according to the documentation and create custom plugins to extend the functionality of MCP. Existing plugin examples include text cleaning and format conversion.

Regular Updates

It is recommended to regularly check for updates to the txtai library and adjust the configuration and code according to the new version to maintain the system's efficiency and security.

📖 Tutorials and Support

Learning Resources

• Official Documentation: Details all the functions and usage methods of the MCP server.
• Community Forum: Provides a platform for user communication and technical support.

Problem Feedback

If you encounter any problems, you can submit a bug report through the Issues page or seek help in the community.