This project provides a native macOS implementation for AI agents to interact with a computer, forking Anthropic's original Docker-based approach. It enables autonomous control of macOS via GUI, screen capture, and file system manipulation, targeting developers and power users seeking to integrate LLMs with their desktop environment.

How It Works

The system bypasses Docker, leveraging native macOS commands and utilities for direct system interaction. It utilizes cliclick for keyboard and mouse control, native screen capture, and a Streamlit-based GUI. This native approach aims for simpler setup and potentially lower overhead compared to containerized solutions, while supporting multiple LLM providers.

Quick Start & Requirements

• Install: git clone the repo, create/activate a Python 3.12+ virtual environment, run ./setup.sh, then pip install -r requirements.txt.
• Prerequisites: macOS Sonoma 15.7+, Python 3.12+, Homebrew, cliclick (brew install cliclick).
• Setup: Requires an Anthropic API key and configuration in a .env file.
• Demo: Run streamlit run streamlit.py.
• Links: GitHub Repo

Highlighted Details

• Native macOS GUI interaction, eliminating Docker dependency.
• Supports multiple LLM providers (Anthropic, Bedrock, Vertex).
• Screen capture and keyboard/mouse control via cliclick.
• Streamlit interface with automatic screen resolution scaling.

Maintenance & Community

The project is a fork of Anthropic's work, maintained by deedy. No specific community channels or roadmap are detailed in the README.

Licensing & Compatibility

The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

This is a beta feature with inherent security risks, as the AI agent has full control over the Mac. The underlying Beta API is subject to change. Optimal performance is recommended at specific resolutions (e.g., 1024x768), with higher resolutions being scaled down.