Vllm Benchmark

🚀 MCP vLLM Benchmarking Tool

This is a proof-of-concept demonstrating the possibility of using MCP for interactive benchmarking of vLLM.

We're no strangers to benchmarking. Check out our blog: Benchmarking vLLM

This is just an example of exploring the possibilities of MCP.

🚀 Quick Start

💻 Usage Examples

Basic Usage

1. Clone the repository.
2. Add it to your MCP server:

{
    "mcpServers": {
        "mcp-vllm": {
            "command": "uv",
            "args": [
                "run",
                "/Path/TO/mcp-vllm-benchmarking-tool/server.py"
            ]
        }
    }
}

Then, you can prompt like the following example:

Benchmark vllm for this endpoint: http://10.0.101.39:8888
Benchmark the following model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
Run the benchmark 3 times, using 32 prompts each time, then compare the results, but ignore the first iteration as it's a warm-up.

📋 To-Do:

• Due to some random outputs from vllm, it might show that some invalid JSON was found. I haven't really looked into this issue yet.