Mspaint MCP Server V2

🚀 Automating the MSPaint Application with a Google Gemini-based AI Agent

This project demonstrates how to use the Google Gemini model to automate the control of the MSPaint application via a Middleware Control Protocol (MCP) server. The AI agent will receive complex mathematical expressions, calculate and verify the results step by step, and draw the final answer in MSPaint.

🚀 Quick Start

Prerequisites

• Python Version: 3.11+
• It is recommended to use Conda to manage the environment.
• Other dependencies: pywin32, pywinauto, fastmcp, python-dotenv, google-genai, rich

Installation Steps

1. Create a Conda virtual environment:

conda create -n eagenv python=3.11
conda activate eagenv

2. Install the required packages:

pip install -r requirements.txt

3. Configure the Gemini API key: Create a file named .env and add the following content:

GEMINI_API_KEY=your_api_key_here

Usage

1. Start the MCP server:

python mcp_server.py

2. Start the MCP client:

python mcp_client.py

✨ Features

• Utilize the Google Gemini model to automate the MSPaint application.
• Enable the AI agent to handle complex mathematical expressions, perform step - by - step calculations, and verify results.
• Draw the final answer in MSPaint.

📦 Installation

The installation steps are described in the "Quick Start" section.

💻 Usage Examples

Basic Usage

MCP Server (mcp_server.py)

from fastmcp import MCPServer

class MSPaintMCP(MCPServer):
    def __init__(self):
        super().__init__()
        self.add_tool(open_paint)
        self.add_tool(draw_rectangle)
        self.add_tool(add_text_in_paint)
        # Add other tools...

if __name__ == "__main__":
    server = MSPaintMCP()
    server.start()

MCP Client (mcp_client.py)

from fastmcp import MCPClient

client = MCPClient('localhost', 5000)
client.send_goal("solve_math_expression", {
    'expression': '123 + 456 * 789',
    'draw_result': True
})

📚 Documentation

Project Architecture

MCP Server (mcp_server.py)

• Main Functions: Define tools for interacting with MSPaint (e.g., open_paint, draw_rectangle, add_text_in_paint) and various mathematical operation tools (e.g., add, subtract, multiply, divide, verify_calculation, verify_consistency).
• Technical Highlights:
  • Use the pywinauto library to automate the control of the MSPaint application.
  • Expose these tools to external clients through the fastmcp library.

MCP Client (mcp_client.py)

• Main Functions: Connect to the MCP server, use the Google Gemini model to generate instructions, and solve mathematical expressions.
• Implementation Details:
  • Set system prompts to define the AI's behavior boundaries and available tools.
  • Parse the model output to determine the tools to be called and their parameters.
  • Process tool responses and feed them back to the model for the next operation.

Core Components

System Prompts (Provided by mcp_client.py)

System prompts define the AI's behavior boundaries and available tools. Here is a simplified example:

• System Prompt Content:
  • Define the callable tools and their functions.
  • Specify the steps for solving mathematical expressions, including step - by - step calculation and verification.

Available Tools

MSPaint Control Tools

1. open_paint(): Open the MSPaint application.
2. draw_rectangle(x1, y1, x2, y2): Draw a rectangle at the specified coordinates.
3. add_text_in_paint(text, x, y): Add text at the specified position.

Mathematical Operation Tools

1. add(a, b): Return the sum of a and b.
2. subtract(a, b): Return the result of a minus b.
3. multiply(a, b): Return the product of a and b.
4. divide(a, b): Return the result of a divided by b, and handle the division - by - zero exception.

Verification Tools

1. verify_calculation(a, b, expected_result): Verify whether the calculation of a and b is consistent with the expected result.
2. verify_consistency(steps): Ensure the correctness of all calculation steps.

🔧 Technical Details

• The pywinauto library is used to automate the MSPaint application, which provides a convenient way to interact with Windows GUI applications.
• The fastmcp library is used to expose tools to external clients, enabling communication between the server and the client.

📄 Notes

⚠️ Important Note

• Error Handling: Ensure that all tool calls include appropriate exception handling.
• Performance Optimization: Optimize the efficiency of graphic drawing and mathematical calculations as needed.

🤝 Contribution Guidelines

Welcome to submit issue reports and feature suggestions. For more details, please refer to the project repository.