Shared Knowledge RAG

🚀 Shared Knowledge Base

Integrate Large Language Models (LLMs) with local knowledge bases to achieve smarter information retrieval and processing capabilities.

🚀 Quick Start

This system integrates Large Language Models (LLMs) with local knowledge bases, offering enhanced information retrieval and processing capabilities. It automatically loads Markdown and text files, converts them into vectors, and enables intelligent retrieval based on vector similarity.

✨ Features

Core Functions

1. Knowledge Base Construction: Automatically load Markdown files (.md, .mdx) and text files (.txt) from a specified directory and chunk their contents.
2. Vectorized Storage: Use the OpenAI API to convert text data into high - dimensional vectors and support multiple vector storage solutions.
3. Intelligent Retrieval: Quickly locate the most relevant document fragments to the query through vector similarity calculation.

Enhanced Retrieval Features

The system provides rich result information, including but not limited to:

• Location Marking: Precise content positioning at the paragraph level.
• Document Type: Support the identification of multiple formats such as Markdown and text files.
• Content Summary: Automatically generate a concise content overview.
• Keyword Extraction: Intelligently extract core theme words.

An example output format is as follows:

{
  "hits": [
    {
      "id": "chunk_123",
      "score": 0.95,
      "content": "This is an example text content...",
      "position": [42, 56],
      "document_type": "Markdown",
      "summary": "This text mainly tells about...",
      "keywords": ["Keyword 1", "Keyword 2"]
    }
  ]
}

These extended information significantly improve the LLM's efficiency in understanding and utilizing retrieval results.

🔧 Technical Details

Technical Architecture

The system adopts a modular design, and its core components include:

1. Knowledge Base Manager: Responsible for document loading, chunking, and vectorization processing.
2. Vector Storage Engine: Support multiple storage solutions for flexible expansion.
3. Retrieval Service: Conduct efficient queries based on vector similarity.

Supported Vector Storage Solutions

• HNSWLib (default): A lightweight local storage solution.
• Chroma: An open - source vector database.
• Pinecone: A hosted vector service (API key required).
• Milvus: A distributed vector retrieval engine.
• Weaviate: A schema - first knowledge graph storage (Docker environment required).

📦 Installation

Environment Configuration

Configure the system through the following environment variables:

Usage Examples

HNSWLib (Default)

No additional environment preparation is required. Run directly:

./rebuild-vector-store-hnsw.sh

Weaviate

1. Start the Docker container:

./start-weaviate.sh

2. Rebuild the vector storage:

./rebuild-vector-store-weaviate.sh

3. Check the service status:

curl http://localhost:8080/v1/.well-known/ready

4. Stop the service:

docker-compose down

🤝 Contribution

The community is welcome to participate in improvement and optimization. Please refer to [CONTRIBUTION.md] for specific contribution guidelines.

📄 License

This project follows the [Apache License 2.0] protocol.