Root Signals MCP

🚀 Model Context Protocol (MCP) Documentation

Model Context Protocol (MCP) is a tool designed for evaluating the quality of AI-generated content. Users can leverage this protocol in the following ways:

1. Directly Invoke MCP Service: Install and run the provided MCP server for local or cloud-based usage.
2. Integrate into Applications: Incorporate the MCP client into your Python applications to achieve automated evaluation.

✨ Features

• Evaluators List: Offers a variety of predefined evaluation metrics, covering multiple dimensions such as accuracy, clarity, and relevance.
• Evaluation Interfaces: Supports four evaluation methods:
  • ID-based Evaluation: Evaluate content using a specified evaluator ID.
  • Name-based Evaluation: Evaluate content using the evaluator's name.
  • Retrieval-based Generation Evaluation: Evaluate content in conjunction with an external knowledge base context.
  • Name-based Retrieval Generation Evaluation: Evaluate content by combining context and the evaluator's name.

📦 Installation

# Clone the repository
git clone https://github.com/your-repository.git
cd your-repository

# Install dependencies
pip install -r requirements.txt

# Start the service
uv run server .

💻 Usage Examples

Basic Usage

import os
from root_mcp_server.client import RootSignalsMCPClient

async def main():
    mcp_client = RootSignalsMCPClient()
    
    try:
        await mcp_client.connect()
        
        # Get the list of all evaluators
        evaluators = await mcp_client.list_evaluators()
        print(f"Detected {len(evaluators)} evaluators")
        
        # Evaluate using ID
        result_id = await mcp_client.run_evaluation(
            evaluator_id="eval-123456789",
            request="What is the capital of France?",
            response="Paris is the capital of France."
        )
        print(f"Score from ID-based evaluation: {result_id['score']}")
        
        # Evaluate using name
        result_name = await mcp_client.run_evaluation_by_name(
            evaluator_name="Clarity",
            request="What is the capital of France?",
            response="Paris is the capital of France."
        )
        print(f"Score from name-based evaluation: {result_name['score']}")
        
    finally:
        await mcp_client.disconnect()

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Advanced Usage

Users can directly input the following command in Cursor to evaluate the prompt templates of generative AI:

Evaluate the summarizer prompt in terms of clarity and precision. Use RootSignals.

🤝 Contribution Guidelines

How to Contribute

We welcome any contributions that enhance the user experience. Here are the steps:

1. Synchronize the Codebase: Use the uv sync --extra dev command to fetch the latest code and dependencies.
2. Install Pre-commit Hooks: Run pre-commit install to ensure code is automatically formatted before each commit.
3. Add Features or Fix Issues: Add new code and test cases to the src/root_mcp_server/tests/ directory.
4. Build and Test: Run docker compose up --build to start the service. Set the environment variable ROOT_SIGNALS_API_KEY=<your_key> to execute unit tests and ensure all tests pass.
5. Format the Code: Use the ruff format . && ruff check . command to maintain consistent code style.

Notes

• Create a separate branch for new features or issue fixes and add detailed commit messages before submission.
• If you encounter any issues, please submit an Issue or a pull request promptly.

⚠️ Known Limitations

Functional Limitations

• Evaluation Scope: The current version only supports the evaluation of text-generated content and does not cover other forms such as images and audio.
• Context Size: The retrieval-based generation evaluation feature currently only supports small-scale knowledge bases.

Performance Limitations

• Processing Latency: Large-scale data or multi-round dialogue scenarios may lead to increased evaluation time.

📫 Feedback

If you encounter any issues during use, please provide feedback through the following channels:

• Submit an Issue
• Send an email to support@yourdomain.com

⚠️ Important Note

The content of this documentation may be updated at any time. It is recommended to check regularly for the latest information.