Atla MCP Server

🚀 Atla MCP Server

An MCP server implementation that provides standardized interfaces, enabling Large Language Models (LLMs) to interact with the Atla API for state-of-the-art LLM evaluations.

For more information, visit here. To learn more about the Model Context Protocol, visit here.

✨ Features

• evaluate_llm_response: Evaluate the response of a large language model to a prompt based on given evaluation criteria. This function uses the underlying Atla evaluation model and returns a dictionary containing the score of the model's response and text feedback (including opinions on the model's response).
• evaluate_llm_response_on_multiple_criteria: Evaluate the response of a large language model to a prompt from the perspective of multiple evaluation criteria. This function uses the underlying Atla evaluation model and returns a list where each element is a dictionary containing the evaluation score and feedback for a certain criterion.

🚀 Quick Start

To use the MCP server, you need an Atla API key. You can find an existing key here or create a new one here.

📦 Installation

We recommend using uv to manage your Python environment. Refer to here for installation instructions.

Manually Run the Server

After installing uv and obtaining an Atla API key, you can manually run the MCP server:

ATLA_API_KEY=<your-api-key> uvx atla-mcp-server

Connect to the Server

Encountered issues or need help connecting to other clients? Feel free to raise an issue or contact us! Here.

OpenAI Agent SDK

For more details on using the OpenAI Agent SDK to interact with the MCP server, refer to the official documentation.

1. Install the OpenAI Agent SDK:

pip install openai-agents

2. Connect to the server using the OpenAI Agent SDK:

import os

from agents import Agent
from agents.mcp import MCPServerStdio

async with MCPServerStdio(
        params={
            "command": "uvx",
            "args": ["atla-mcp-server"],
            "env": {"ATLA_API_KEY": os.environ.get("ATLA_API_KEY")}
        }
    ) as atla_mcp_server:
    ...

Claude Desktop

For more details on configuring the MCP server in Claude Desktop, refer to the official quick start guide.

1. Add the following content to the claude_desktop_config.json file:

{
  "mcpServers": {
    "atla-mcp-server": {
      "command": "uvx",
      "args": ["atla-mcp-server"],
      "env": {
        "ATLA_API_KEY": "<your-atla-api-key>"
      }
    }
  }
}

2. Restart Claude Desktop to apply the changes.

Now, you should see the atla-mcp-server option in the list of available MCP tools.

Cursor

For more details on configuring the MCP server in Cursor, refer to the official documentation.

1. Add the following content to the ~/.config/cursor.json file:

{
  "mcpServers": {
    "atla-mcp-server": {
      "command": "uvx",
      "args": ["atla-mcp-server"],
      "env": {
        "ATLA_API_KEY": "<your-atla-api-key>"
      }
    }
  }
}

🤝 Contribution

Contributions to this project are welcome! Visit here to see how you can get involved.

📄 License

This project is licensed under the MIT License.