Deepspringai Parquet MCP Server

🚀 Parquet MCP Server

The Parquet MCP Server is a powerful tool designed to handle various operations on Parquet files, such as generating embeddings, retrieving file information, and converting to different database formats. It also provides a way to process Markdown files and convert them into structured Parquet files.

🚀 Quick Start

📦 Installation

Project Structure

parquet-mcp-server/
├── README.md                # Project description file
├── src/                    # Source code directory
│   ├── __init__.py         # Package initialization file
│   ├── client.py           # MCP client interface
│   └── tools/              # Tool module
│       ├── embedding.py    # Embedding tool
│       ├── parquet_info.py  # Parquet file information tool
│       ├── duckdb_conversion.py  # DuckDB conversion tool
│       ├── postgres_conversion.py  # PostgreSQL conversion tool
│       └── markdown_processing.py  # Markdown processing tool
├── tests/                  # Test case directory
│   ├── __init__.py         # Test package initialization file
│   ├── test_embedding.py    # Embedding function test
│   ├── test_parquet_info.py  # Parquet file information tool test
│   ├── test_duckdb_conversion.py  # DuckDB conversion test
│   ├── test_postgres_conversion.py  # PostgreSQL conversion test
│   └── test_markdown_processing.py  # Markdown processing test
└── requirements.txt        # Project dependency management file

Environment Variables

Add the following environment variables to the .env file:

# Required configuration
OPENAI_API_KEY=your_openai_api_key  # OpenAI API key

# Optional configuration (adjust as needed)
OLLAMA_SERVER_URL=http://localhost:11434  # Ollama server address, default is local

Install and Run

Using Command Line Tools

pip install -r requirements.txt  # Install dependencies
python src/run_server.py  # Start the MCP server

Using the Test Client

from parquet_mcp_server.client import (
    convert_to_duckdb, 
    embed_parquet, 
    get_parquet_info, 
    convert_to_postgres,
    process_markdown_file
)

# Test DuckDB conversion
result = convert_to_duckdb(
    parquet_path="input.parquet",
    output_dir="db_output"
)

# Test embedding function
result = embed_parquet(
    input_path="input.parquet",
    output_path="output.parquet", 
    column_name="text",
    embedding_column="embeddings",
    batch_size=2
)

# Test Parquet information tool
result = get_parquet_info("input.parquet")

# Test PostgreSQL conversion
result = convert_to_postgres(
    parquet_path="input.parquet", 
    table_name="my_table"
)

# Test Markdown processing
result = process_markdown_file(
    file_path="input.md",
    output_path="output.parquet"
)

✨ Features

💻 Usage Examples

Basic Usage

# The following are basic usage examples of each tool
# Embedding tool
def embed_parquet(
    input_path: str,
    output_path: str, 
    column_name: str = "text",
    embedding_column: str = "embeddings", 
    batch_size: int = 16
) -> None:
    # Function: Generate embedding vectors for the specified column and save them to a new column
    # Input:
    #   input_path: Path to the input Parquet file
    #   output_path: Path to the output Parquet file
    #   column_name: Name of the column to generate embeddings for, default is "text"
    #   embedding_column: Name of the column to save the embedding results, default is "embeddings"
    #   batch_size: Batch size for processing, default is 16

# Parquet file information tool
def get_parquet_info(parquet_path: str) -> Dict:
    # Function: Get basic information and statistical information of the Parquet file
    # Input:
    #   parquet_path: Path to the Parquet file
    # Output:
    #   Dict: A dictionary containing file size, number of rows, column information, etc.

# DuckDB conversion tool
def convert_to_duckdb(
    parquet_path: str, 
    output_path: str,
    table_name: str = "parquet_table"
) -> None:
    # Function: Convert the Parquet file to a DuckDB table file
    # Input:
    #   parquet_path: Path to the input Parquet file
    #   output_path: Path to the output .db file
    #   table_name: Name of the target table, default is "parquet_table"

# PostgreSQL conversion tool
def convert_to_postgres(
    parquet_path: str,
    host: str,
    database: str,
    user: str,
    password: str,
    table_name: str = "parquet_table",
    chunk_size: int = 1000
) -> None:
    # Function: Import the data from the Parquet file into a PostgreSQL database table
    # Input:
    #   parquet_path: Path to the input Parquet file
    #   host: Database host address
    #   database: Database name
    #   user: Database username
    #   password: Database password
    #   table_name: Name of the target table, default is "parquet_table"
    #   chunk_size: Amount of data to insert each time, default is 1000

# Markdown processing tool
def process_markdown_file(
    file_path: str,
    output_path: str,
    encoding: str = "utf-8",
    split_by: str = None,
    keep_separators: bool = False
) -> None:
    # Function: Convert the Markdown file to a structured Parquet file
    # Input:
    #   file_path: Path to the input Markdown file
    #   output_path: Path to the output Parquet file
    #   encoding: Character encoding, default is "utf-8"
    #   split_by: Split the content by a separator (optional)
    #   keep_separators: Whether to keep the separators, default is False

🧪 Testing

Start the Test Server

python src/run_server.py  # Start the MCP server

Execute Test Cases

python -m pytest tests/  # Run all test cases
python -m pytest tests/test_embedding.py  # Test the embedding function
python -m pytest tests/test_parquet_info.py  # Test the Parquet file information tool
python -m pytest tests/test_duckdb_conversion.py  # Test the DuckDB conversion tool
python -m pytest tests/test_postgres_conversion.py  # Test the PostgreSQL conversion tool
python -m pytest tests/test_markdown_processing.py  # Test the Markdown processing tool

📚 Documentation

API Response Format

Embedding Response

{
    "status": "success",
    "message": "embedding completed successfully",
    "result": {
        "input_count": 100,
        "output_file": "output.parquet"
    }
}

File Information Response

{
    "file_size": "12.5 MB",
    "num_rows": 1000,
    "columns": [
        {"name": "text", "type": "string"},
        {"name": "label", "type": "int"}
    ]
}

⚠️ Important Note

• Ensure that all dependencies are installed, especially llama-cpp-python and duckdb.
• Set the correct environment variables to avoid API call errors.
• When processing large files, it is recommended to process them in batches to ensure performance.