Omniparser AutoGUI

This is an MCP server based on OmniParser that can analyze screen content and automatically operate the GUI interface, mainly running on the Windows system.

Operating system automation Developer tools #Screen Analysis #Automated Operation #MCP Protocol #Windows .Python

rating : 2.5 points

downloads : 10.7K

update time : 2025-04-28

Open Site

What is OmniParser Automated GUI MCP?

This is an intelligent server that can 'see' and interact with your computer screen. It uses advanced AI (OmniParser) to understand the content displayed on the screen and can automatically perform GUI operations, such as clicking buttons or entering text.

How does it work?

The server connects to applications that support MCP (such as ClaudeChat). It achieves automated tasks by analyzing the screen content and generating appropriate operation instructions based on the context.

Why choose OmniParser Automated GUI MCP?

Compared with traditional scripts, OmniParser provides more powerful and flexible screen analysis capabilities. It can handle complex UI elements and provide intelligent operations through context understanding.

Features

Multi - language Support

By setting the OCR_LANG environment variable, it supports text recognition in multiple languages.

Window Target Location

Use the TARGET_WINDOW_NAME environment variable to specify the specific window to control.

Context Understanding

Generate intelligent operation instructions based on the screen content and context to improve the accuracy of automated tasks.

Frequently Asked Questions

Does it support Mac or Linux?

Can I use different languages for text recognition?

How do I specify the window to control?

Related Resources

OmniParser GitHub

The core AI technology for screen analysis.

Model Context Protocol Documentation

The official documentation of the MCP protocol.

LibreChat Example Integration

An example client code repository for use with this server.

🚀 omniparser-autogui-mcp

This is an MCP server that uses OmniParser to analyze the screen and perform automatic GUI operations. It is confirmed to be available on Windows.

🚀 Quick Start

This is an MCP server that uses OmniParser to analyze the screen and perform automatic GUI operations. It has been confirmed to work on Windows. (Japanese version here)

📄 License

This project is licensed under the MIT license, excluding sub - modules and sub - packages. The OmniParser repository uses the CC - BY - 4.0 license. Each OmniParser model has a different license (Reference).

📦 Installation

Please follow these steps:

git clone --recursive https://github.com/NON906/omniparser-autogui-mcp.git
cd omniparser-autogui-mcp
uv sync
set OCR_LANG=en
uv run download_models.py

(On non - Windows systems, use export instead of set.) (If you want langchain_example.py to run correctly, use uv sync --extra langchain.)

Add the following content to your claude_desktop_config.json file:

{
  "mcpServers": {
    "omniparser_autogui_mcp": {
      "command": "uv",
      "args": [
        "--directory",
        "D:\\CLONED_PATH\\omniparser-autogui-mcp",
        "run",
        "omniparser-autogui-mcp"
      ],
      "env": {
        "PYTHONIOENCODING": "utf-8",
        "OCR_LANG": "en"
      }
    }
  }
}

(Replace D:\\CLONED_PATH\\omniparser-autogui-mcp with the actual path where you cloned the repository.)

The env allows the following additional configurations:

OMNI_PARSER_BACKEND_LOAD Specify 1 if there is a conflict with other clients (e.g., LibreChat).
TARGET_WINDOW_NAME Specify the window name if you want to target a specific window. If not specified, the entire screen will be processed by default.
OMNI_PARSER_SERVER Specify the server address and port (e.g., 127.0.0.1:8000) if you want OmniParser processing to be performed on another device. The server can be started with uv run omniparserserver.
SSE_HOST, SSE_PORT If specified, communication will be via SSE instead of standard input/output.
SOM_MODEL_PATH, CAPTION_MODEL_NAME, CAPTION_MODEL_PATH, OMNI_PARSER_DEVICE, BOX_TRESHOLD These are configurations for OmniParser. Normally, manual configuration is not required.