This project provides a self-hosted desktop application that allows AI models, specifically Anthropic's Claude, to control a computer's mouse and keyboard. It's designed for users who want to experiment with AI-driven automation for tasks like browsing, coding, and general computer interaction, offering a novel way to delegate tasks to an AI.

How It Works

The application leverages Anthropic's computer use capability, enabling Claude to interpret natural language commands and execute them via simulated mouse and keyboard inputs. It uses Python with the PyQt framework for the desktop interface and relies on screenshots to understand the current UI state for accurate interaction.

Quick Start & Requirements

• Install via pip install -r requirements.txt after setting up a Python 3.10+ virtual environment.
• Requires an Anthropic API key, set in a .env file.
• Recommended to install Firefox for improved UI detection.
• Official quick-start and setup instructions are available in the README.

Highlighted Details

• Enables AI to perform arbitrary tasks controllable by mouse and keyboard.
• Supports macOS, Windows, and Linux platforms where Python runs.
• Includes productivity keybindings for controlling agent actions.
• Acknowledges that AI computer control is currently slow and experimental.

Maintenance & Community

The project is a recent creation, with the author open to contributions and ideas via Twitter (@ishanxnagpal). A formal roadmap is not yet established.

Licensing & Compatibility

Licensed under the Apache License 2.0, permitting commercial use and integration with closed-source projects.

Limitations & Caveats

This is experimental software with potential for unintended actions. The AI may occasionally fail to take screenshots for validation, leading to incorrect input placement. Users are advised to monitor actions closely and be aware that Anthropic can view screenshots during operation.