Prompt Tester

🚀 MCP Prompt Framework Documentation

This documentation provides a comprehensive guide to the MCP prompt framework, including environment configuration, code examples for prompt testing, and instructions for integrating MCP with agents.

🚀 Quick Start

Environment Configuration

Required Environment Variables:

• OPENAI_API_KEY - Your OpenAI API key.
• ANTHROPIC_API_KEY - Your Anthropic API key.

Optional Langfuse Tracking Settings (for logging and observability):

• LANGFUSE_SECRET_KEY - Your Langfuse secret key.
• LANGFUSE_PUBLIC_KEY - Your Langfuse public key.
• LANGFUSE_HOST - The URL of your Langfuse instance.

If you don't need to use the Langfuse tracking feature, you can leave these configurations blank.

💻 Usage Examples

Complete Code Example for Prompt Testing with the MCP Framework

import asyncio
import json
from mcp.client.session import ClientSession
from mcp.client.stdio import StdioServerParameters, stdio_client

async def main():
    async with stdio_client(
        StdioServerParameters(command="prompt-tester")
    ) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            
            # 1. List all available providers and models
            providers_result = await session.call_tool("list_providers", {})
            print("Available providers and models:", providers_result)
            
            # 2. Perform a basic test with a single model
            comparison_result = await session.call_tool("test_comparison", {
                "comparisons": [
                    {
                        "provider": "openai",
                        "model": "gpt-4",
                        "system_prompt": "You are a helpful assistant.",
                        "user_prompt": "Explain quantum computing in simple terms.",
                        "temperature": 0.7,
                        "max_tokens": 500
                    }
                ]
            })
            print("Single model test result:", comparison_result)
            
            # 3. Compare multiple prompts/models
            comparison_result = await session.call_tool("test_comparison", {
                "comparisons": [
                    {
                        "provider": "openai",
                        "model": "gpt-4",
                        "system_prompt": "You are a helpful assistant.",
                        "user_prompt": "Explain quantum computing in simple terms.",
                        "temperature": 0.7
                    },
                    {
                        "provider": "anthropic",
                        "model": "claude-3-opus-20240229",
                        "system_prompt": "You are a helpful assistant.",
                        "user_prompt": "Explain quantum computing in simple terms.",
                        "temperature": 0.7
                    }
                ]
            })
            print("Comparison test result:", comparison_result)
            
            # 4. Start a multi-turn conversation
            conversation_start = await session.call_tool("test_multiturn_conversation", {
                "mode": "start",
                "provider": "openai",
                "model": "gpt-4",
                "system_prompt": "You are an expert in physics who can explain complex concepts in simple terms.",
                "user_prompt": "What is dark matter?"
            })
            print("Conversation started:", conversation_start)
            
            # Get the conversation ID from the response
            response_data = json.loads(conversation_start.text)
            conversation_id = response_data.get("conversation_id")
            
            # Continue the conversation
            if conversation_id:
                conversation_continue = await session.call_tool("test_multiturn_conversation", {
                    "mode": "continue",
                    "conversation_id": conversation_id,
                    "user_prompt": "How does that relate to dark energy?"
                })
                print("Continuing the conversation:", conversation_continue)
                
                # Get the conversation history
                conversation_history = await session.call_tool("test_multiturn_conversation", {
                    "mode": "get",
                    "conversation_id": conversation_id
                })
                print("Conversation history:", conversation_history)

asyncio.run(main())

📚 Documentation

MCP Agent Integration

For agents powered by MCP, the integration steps are as follows:

1. Service Discovery: Agents can call the list_providers tool to discover available models and their capabilities.
2. Simple Testing: For quick testing, use the test_comparison tool with a single configuration.
3. Comparison Testing: When an agent needs to evaluate different prompts or models, it can configure multiple options simultaneously using the test_comparison tool.
4. Stateful Interaction: For multi-turn conversations, agents can use the test_multiturn_conversation tool to manage context and conversation history.

This integration method enables agents to:

• Test different prompt variants to find the most effective expression.
• Compare the performance of different models on specific tasks.
• Maintain context information for conversations.
• Adjust temperature parameters to control the creativity and accuracy of generated text.

Through these features, the MCP prompt framework provides a powerful toolset for agents to efficiently design prompts and call models.