MCP Autotest

🚀 MCP Protocol Automated Testing Tool

The MCP Protocol (Model Context Protocol) is a communication protocol for model context interaction. mcp-autotest is an automated testing tool designed specifically for testing MCP protocol implementations. It helps developers quickly verify whether their services comply with the MCP protocol specifications.

🚀 Quick Start

📦 Installation

npx mcp-autotest

💻 Usage Examples

Basic Usage

1. Create Test Cases Edit the YAML test files in the testdata folder:

case: List tools
in_list_tools: {"jsonrpc":"2.0","method":"tools/list","id":1}
out_list_tools: {"jsonrpc":"2.0","result":{"tools":[ { "name": !!re "list-[a-z]+" } ]},"id":1}

2. Execute Tests Run the following command to start the tests:

npx mcp-autotest run testdata -- npx -y @modelcontextprotocol/server-postgres localhost:5432

Advanced Usage

🔧 Dynamic Matching

In some cases, you may need to handle dynamic values (such as timestamps or UUIDs). You can use regular expressions for matching. The tool supports two modes:

1. Full Regular Expressions Use the !!re tag to treat the entire string as a regular expression.

out_list_tools: {"jsonrpc":"2.0","result":{"tools":[ { "name": !!re "list-[a-z]+" } ]},"id":1}

2. Embedded Regular Expressions Use the !!ere tag to embed regular expression fragments in a string.

out_list_tools: {"jsonrpc":"2.0","result":{"tools":[ { "name": !!ere "list-/[a-z]+/-tool" } ]},"id":1}

⚠️ Important Note

• Characters other than the regular expression part need to be escaped to their actual values (using \\).
• Regular expression matching is case-insensitive.

📚 Documentation

Example of Testing PostgreSQL Service

Refer to the PostgreSQL example in the project to learn how to write test cases for specific services.

Output Example

2025/03/31 22:23:39 The actual JSON does not match the expectation.
Got: '{"id":1,"jsonrpc":"2.0","result":{"tools":[{"description":"Run a read-only SQL query","inputSchema":{"properties":{"sql":{"type":"string"}},"type":"object"},"name":"query"}]}}'
Compared with the expectation:
  "result": {
    "tools": {
      "0": {
        "description": {
        ^ The string values do not match.
Expected string: 'Run a read-only SQL query 2'
Actual string: 'Run a read-only SQL query'
Test failed

Example of Dynamic Matching

Suppose the description field returned by the tool is a dynamic value. You can write test cases in the following way:

case: List tools
in_list_tools: {"jsonrpc":"2.0","method":"tools/list","id":1}
out_list_tools: {"jsonrpc":"2.0","result":{"tools":[ { "name": !!ere "list-/[a-z]+/-tool" } ]},"id":1}

In this example:

• "list-" and "-tool" are fixed strings.
• /[a-z]+/ is an embedded regular expression that matches any sequence of lowercase letters.

This can avoid test failures caused by dynamic values such as version numbers.

📄 Summary

mcp-autotest is a powerful automated testing tool that helps developers efficiently verify MCP protocol implementations. With flexible configuration and regular expression support, you can easily meet testing requirements in various scenarios.