Summary

Ghost OS enables AI agents to interact with any native macOS application, overcoming the limitation of chat-based interfaces. It allows agents to perform complex tasks like coding, file manipulation, and email sending directly within desktop apps. Targeting AI developers and power users, it offers self-learning workflows and native interaction without relying on screenshots, extending AI capabilities to the full desktop environment.

How It Works

The core approach utilizes the macOS accessibility tree for structured UI element data, providing precise interaction details. For web apps or dynamic content where the tree is insufficient, a local vision model (ShowUI-2B) provides visual grounding. This dual approach ensures robust interaction with any app. A key innovation is its self-learning workflow system, where complex tasks are discovered once by a frontier model and then efficiently executed by a smaller model, saved as transparent JSON "recipes."

Quick Start & Requirements

Installation is via Homebrew: brew install ghostwright/ghost-os/ghost-os, followed by ghost setup for permissions, MCP, recipes, and vision model configuration. Manual installation is an alternative for macOS developer betas. Prerequisites include macOS (14+ recommended for source builds), Swift 6.2+ (for source builds), and ~3GB for the local vision model.

Highlighted Details

• Offers 26 tools for comprehensive UI interaction, including advanced actions like annotation, hover, long-press, and drag.
• Features self-learning workflows saved as transparent, auditable JSON recipes for efficient task execution.
• Supports interaction with any native macOS application, extending beyond web browsers.
• Processes data locally, ensuring user privacy.
• Integrates with AI agents via the MCP protocol, compatible with clients like Claude Code and Cursor.

Maintenance & Community

The project shows strong community engagement with over 300 stars. Contribution guidelines are available in CONTRIBUTING.md.

Licensing & Compatibility

Released under the permissive MIT license, suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

Exclusively for macOS. Users on macOS developer betas may require manual installation. Building from source needs macOS 14+ and Swift 6.2+. The local vision model requires ~3GB of disk space.