Mcpbench

🚀 🦊 MCPBench: A Benchmark for Evaluating MCP Servers

MCPBench is a framework designed for evaluating MCP servers. It supports two types of servers: web search and database query, and is compatible with both local and remote MCP servers. This framework primarily evaluates different MCP servers (such as Brave Search, DuckDuckGo, etc.) in terms of task completion accuracy, latency, and token consumption under the same LLM and agent configurations. Here is the evaluation report.

The implementation is based on LangProBe: Language Program Benchmark.

🚀 Quick Start

📋 News

We have released the MCPBench framework, along with detailed documentation and experimental reports. For more information, please visit the GitHub repository.

📦 Installation

To install MCPBench, follow these steps:

1. Clone this repository:

git clone https://github.com/modelscope/MCPBench.git

2. Navigate to the project directory:

cd MCPBench

3. Install the dependencies:

pip install -r requirements.txt

💻 Usage Examples

Basic Usage

Here are the basic steps to use MCPBench:

1. Download the pre-trained LLM model.
2. Configure the MCP server.
3. Run the evaluation script.

from mcpbench import *

# Initialize the MCP server
mcp_server = init_mcp_server()

# Load the LLM model
llm_model = load_llm_model()

# Perform the evaluation
results = evaluate(mcp_server, llm_model)

# Output the results
print(results)

📋 Data Format

We support the following data formats:

JSON Format

{
  "unique_id": "",
  "Prompt": "",
  "Answer": ""
}

CSV Format

unique_id,Prompt,Answer
,...(more rows)

📜 Citation

If you find our work helpful, please consider citing our project:

@misc{mcpbench,
  title={MCPBench: A Benchmark for Evaluating MCP Servers},
  author={Zhiling Luo, Xiaorong Shi, Xuanrui Lin, Jinyang Gao},
  howpublished = {\url{https://github.com/modelscope/MCPBench}},
  year={2025}
}

Or refer to our report:

@article{mcpbench_report,
      title={Evaluation Report on MCP Servers}, 
      author={Zhiling Luo, Xiaorong Shi, Xuanrui Lin, Jinyang Gao},
      year={2025},
      journal={arXiv preprint arXiv:2504.11094},
      url={https://arxiv.org/abs/2504.11094},
      primaryClass={cs.AI}
}