Gives AI agents the ability to observe and act on iOS Simulator and Android emulator/device screens, enabling automated mobile development workflows. It targets AI agents and developers building agentic loops, providing a crucial bridge for AI-driven UI interaction and verification.

How It Works

This cross-platform CLI leverages platform-specific Accessibility APIs (Apple's Accessibility APIs, iOS Simulator HID pipeline, Android's AccessibilityService) to provide a unified command surface. It translates screen content into a token-efficient outline format for LLM reasoning and allows direct interaction via alias-cached tap commands. The core loop involves observing the UI, acting on elements, and verifying results, designed for rapid, automated cycles.

Quick Start & Requirements

• Installation: Recommended via Homebrew: brew tap lycorp-jp/tap && brew install lycorp-jp/tap/sim-use.
• Build from Source: Requires macOS 14+ and the latest Xcode toolchain. Build Meta's idb XCFrameworks using ./scripts/build.sh dev, then build sim-use with make build.
• Agent Skill: Install bundled agent skills with sim-use init [--client claude].
• Android Setup: Requires sim-use android init --device <serial> and AGENTS.md for toolchain setup.

Highlighted Details

• Token-Efficient UI Representation: The ui command outputs a compact outline (~16x smaller than raw accessibility trees), suitable for LLM processing.
• AI-Native Design: Features alias-cached taps (@N), structured JSON output with actionable hints, and a bundled agent skill for seamless integration into AI agent loops.
• Performance: Achieves ~300ms observe-act round trips after initial background daemon setup.
• Cross-Platform Consistency: A single command surface operates identically across iOS Simulators and Android emulators/devices.
• Comprehensive Interaction: Supports tap, swipe, type, IME-safe Unicode paste, gestures, hardware buttons, and key combinations.
• Bundled Viewer: Includes a local web application for visualizing the accessibility tree and element interactions.

Maintenance & Community

No specific details regarding maintainers, community channels (like Discord/Slack), or roadmap were found in the provided README.

Licensing & Compatibility

Licensed under the Apache License, Version 2.0. It originated as a fork of cameroncooke/AXe (MIT) and links against Meta's idb (MIT). These permissive licenses allow for commercial use and integration into closed-source projects.

Limitations & Caveats

iOS 16+ introduces an "Allow Paste" prompt requiring interactive approval for the first paste operation per app session. Direct HID keycode injection is iOS-only; Android text input relies on type or paste commands. The background daemon automatically exits after 600 seconds of inactivity.