MCP Evals

🚀 MCP Evaluation Tool

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure that your MCP server tools run correctly and efficiently.

🚀 Quick Start

This tool can be used as a Node.js package or a GitHub Action to evaluate MCP tool implementations.

✨ Features

• Evaluate MCP tool implementations using LLM-based scoring.
• Provide detailed evaluation results including accuracy, completeness, relevance, clarity, and reasoning.
• Can be integrated into your development workflow as a Node.js package or a GitHub Action.

📦 Installation

As a Node.js Package

npm install mcp-evals

As a GitHub Action

Add the following to your workflow file:

name: Run MCP Evaluation
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          
      - name: Install dependencies
        run: npm install
        
      - name: Run MCP Evaluation
        uses: mclenhard/mcp-evals@v1.0.9
        with:
          evals_path: 'src/evals/evals.ts'
          server_path: 'src/index.ts'
          openai_api_key: ${{ secrets.OPENAI_API_KEY }}
          model: 'gpt-4'  # Optional, default is gpt-4

💻 Usage Examples

Basic Usage

1. Create Your Evaluation File

Create a file (e.g., evals.ts) and export your evaluation configuration:

import { EvalConfig } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade, EvalFunction} from "mcp-evals";

const weatherEval: EvalFunction = {
    name: 'Weather Tool Evaluation',
    description: 'Evaluates the accuracy and completeness of weather information retrieval',
    run: async () => {
      const result = await grade(openai("gpt-4"), "What is the weather in New York?");
      return JSON.parse(result);
    }
};
const config: EvalConfig = {
    model: openai("gpt-4"),
    evals: [weatherEval]
  };
  
  export default config;
  
  export const evals = [
    weatherEval,
    // Add other evaluations here
]; 

2. Run the Evaluation

As a Node.js Package

You can run the evaluation using the CLI:

npx mcp-eval path/to/your/evals.ts path/to/your/server.ts

As a GitHub Action

The action will automatically perform the following steps:

1. Run your evaluations.
2. Post a result comment on the PR.
3. Update the comment if the PR is updated.

Advanced Usage

The evaluation results are returned in a structured object. Here is the structure of the evaluation result:

interface EvalResult {
  accuracy: number;        // Score ranges from 1-5
  completeness: number;    // Score ranges from 1-5
  relevance: number;       // Score ranges from 1-5
  clarity: number;         // Score ranges from 1-5
  reasoning: number;       // Score ranges from 1-5
  overall_comments: string; // Summary of strengths and weaknesses
}

📚 Documentation

Configuration

Environment Variables

• OPENAI_API_KEY: Your OpenAI API key (required)

⚠️ Important Note

If you use this GitHub Action for open-source software development, enable data sharing in your OpenAI billing dashboard to claim 2.5 million free GPT-4o mini tokens per day, making this action effectively free to use.

Evaluation Configuration

The EvalConfig interface requires the following:

• model: The language model for evaluation (e.g., GPT-4)
• evals: An array of evaluation functions to run

Each evaluation function must implement the following:

• name: The name of the evaluation
• description: A description of what the evaluation tests
• run: An asynchronous function that takes a model and returns an EvalResult

📄 License

MIT