This repository provides a sample application to learn and build Computer Using Agents (CUAs) with the OpenAI API. It targets developers and researchers looking to integrate AI agents with various computer environments, offering a flexible framework for executing AI-recommended actions.

How It Works

The core of the system involves an agent loop that receives screenshots of a computer interface and recommends actions (e.g., clicks, typing) via the OpenAI API. These actions are then executed in a chosen "computer environment" (local browser, Docker, remote browser), and the resulting screenshots are fed back to the agent. This iterative process allows the AI to interact with and navigate digital environments.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Run CLI: python cli.py --computer local-playwright
• Requires Python 3.x, Playwright SDK (install via prompt on first run).
• Official Docs: Computer use guide

Highlighted Details

• Supports multiple computer environments: LocalPlaywright, Docker, Browserbase, Scrapybara.
• Provides abstractions for Computer (action execution) and Agent (interaction loop).
• CLI tool for easy experimentation with different environments and inputs.
• Example implementations for custom function routing to the computer environment.

Maintenance & Community

• OpenAI maintained.
• Contribution guide available for adding new computer environments.

Licensing & Compatibility

• License: Not explicitly stated in the README.
• Compatibility: Designed for use with OpenAI API; commercial use implications depend on OpenAI's API terms.

Limitations & Caveats

The Computer use feature is in preview, with a caution against using it in authenticated or high-stakes environments due to potential exploits and mistakes. The Docker environment setup requires building and running a container, which can take time on the first run.