Macos Ui Automation MCP

🚀 Playwright MCP for macOS 🎭

Similar to Playwright, but designed for native macOS applications. It enables controlling any Mac app using natural language through Claude, which is ideal for developing and testing Mac applications with AI assistance.

🚀 Quick Start

1. Install

git clone https://github.com/mb-dev/macos-ui-automation-mcp.git
cd macos-ui-automation-mcp
uv sync

2. Set Up Accessibility Permissions

⚠️ Important Note

Enable accessibility for your parent application:

• If using Terminal: Add Terminal to System Settings → Privacy & Security → Accessibility
• If using VS Code: Add VS Code to System Settings → Privacy & Security → Accessibility
• If using Claude Code: Add Claude Code to System Settings → Privacy & Security → Accessibility

The parent app needs permission because it's the one actually executing the MCP server.

3. Configure Claude Code

Add to your Claude Code MCP settings:

{
  "mcpServers": {
    "macos-ui-automation": {
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/macos-ui-automation-mcp",
        "run",
        "macos-ui-automation-mcp"
      ]
    }
  }
}

4. Start Automating!

Now you can ask Claude things like:

• "Find all buttons in the Calculator app"
• "Click the submit button in my app"
• "Click the screenshot button to capture the current window"
• "Test the login flow by filling in credentials and clicking submit"

✨ Features

This is an MCP (Model Context Protocol) server that gives Claude the ability to see and interact with any macOS application - just like Playwright does for web browsers, but for native Mac apps.

Perfect for:

• 🧪 Testing Mac applications - "Test the login flow in my app"
• 🔍 App development - "Check if all buttons are properly labeled"
• 🤖 UI automation - "Fill out this form and submit it"
• 📱 App exploration - "Show me all the interactive elements in Finder"

📦 Installation

The installation steps are included in the "Quick Start" section above.

💻 Usage Examples

Basic Usage

You can use the following commands to interact with macOS applications:

# All buttons in any app
$..[?(@.role=='AXButton')]

# Buttons with specific text
$..[?(@.title=='Submit')]

# All text fields that are enabled
$..[?(@.role=='AXTextField' && @.enabled==true)]

# Elements with accessibility identifiers
$..[?(@.ax_identifier=='loginButton')]

# Elements in a specific app
$.processes[?(@.name=='Calculator')]..[?(@.role=='AXButton')]

Advanced Usage

Test Automation

"Test my login flow:
1. Find the username field and type 'testuser'  
2. Find the password field and type 'password123'
3. Click the login button
4. Verify a success message appears"

UI Validation

"Check my settings window:
- Are all buttons properly labeled?
- Are there any text fields without accessibility identifiers?
- Click the screenshot button to capture the current state"

Accessibility Auditing

"Audit my app for accessibility:
- Find all interactive elements without accessibility labels
- Check if keyboard navigation works properly
- Identify any elements that might be hard to use"

📚 Documentation

Available Tools

Adding Screenshots to Your App

We don't provide built-in screenshot functionality, but you can easily add it to your Mac app! Check out our complete Swift implementation example based on a real-world app.

Key points:

• Uses ScreenCaptureKit (macOS 14+) for high-quality captures
• Automatically finds your app window
• Saves timestamped screenshots to Documents/Screenshots
• Integrates perfectly with this MCP - just add an accessibility identifier!

Usage with Playwright MCP:

"Click the screenshot button to capture the current window"

The MCP will find your button by accessibility ID and trigger the screenshot!

Development Setup

For contributors and advanced users:

# Clone and install
git clone https://github.com/mb-dev/macos-ui-automation-mcp.git
cd macos-ui-automation-mcp
uv sync --dev

# Run tests
uv run python -m pytest tests/ -v

# Check code quality  
uv run ruff check src/ tests/ mcp_server_wrapper.py
uv run ruff format

# Test the MCP server
uv run macos-ui-automation-mcp

🔧 Technical Details

Architecture

Built with:

• FastMCP - MCP server framework
• PyObjC - macOS accessibility API bindings
• Pydantic - Type-safe data models
• JSONPath - Powerful element querying
• Comprehensive test suite - Fake system for testing without real UI

Important Notes

⚠️ Important Note

• Accessibility Permissions: Must be granted to the parent application (Terminal, VS Code, etc.), not to Python or the MCP server itself. Required for any UI automation on macOS.
• Screenshot Permissions: If your app has screenshot functionality, it needs Screen Recording permission. Add your app to System Settings → Privacy & Security → Screen Recording. This is separate from accessibility permissions.
• Performance Tips: Use app-specific searches when possible (find_elements_in_app). Shallow searches are faster for overviews, while deep searches are thorough but slower.
• Limitations: Requires accessibility API access (some apps restrict this). Works best with native macOS applications. Some system-level elements may not be accessible.

📄 License

This project is under the MIT License. You are free to use it in your projects!

🎭 Why "Playwright for Mac"?

Just like Playwright revolutionized web testing by providing a simple API to control browsers, this tool does the same for native macOS applications. Instead of writing complex GUI automation scripts, just tell Claude what you want to test or automate in natural language.

Perfect for the age of AI-assisted development! 🤖

💡 Usage Tip

Need help? Check the examples/ folder or open an issue. Better yet, submit a PR! 😄