AutoDroid enables Large Language Models (LLMs) to automate intelligent tasks on smartphones by leveraging their UI elements and natural language task descriptions. It is designed for researchers and developers working on AI-driven mobile automation and aims to provide a framework for LLM-powered smartphone interaction.

How It Works

AutoDroid builds upon the DroidBot framework, integrating LLMs to interpret tasks and interact with Android applications. It captures UI states (screenshots and view hierarchies) and uses these, along with task descriptions, to prompt an LLM for actions. The LLM's output is then translated into executable commands to navigate and operate the mobile application.

Quick Start & Requirements

• Install via pip: git clone git@github.com:MobileLLM/AutoDroid.git && cd AutoDroid/ && pip install -e .
• Prerequisites: Python, Java, Android SDK (with platform_tools in PATH).
• Requires a GPT API key.
• Tested with apps from Google Cloud (DroidTask dataset) or provided APKs.
• Requires a device or emulator connected via ADB.
• Official Docs: https://github.com/MobileLLM/AutoDroid

Highlighted Details

• Leverages LLMs for intelligent task interpretation and action generation.
• Utilizes the DroidTask dataset, containing app states (screenshots, view hierarchies) and task definitions.
• Implemented based on the DroidBot framework for UI interaction.
• Supports custom APKs for testing.

Maintenance & Community

• Primarily a research project, with contributions welcomed.
• No specific community channels (Discord/Slack) or roadmap mentioned.

Licensing & Compatibility

• License not explicitly stated in the README.
• Intended for research purposes only.

Limitations & Caveats

The current implementation struggles with task completion determination and exhibits unstable automation performance due to LLM randomness and GUI variations. It requires an ADB-connected host machine, not a standalone on-device solution. The project warns of potential unintended actions like account modifications.