Midscene Skills offers AI-powered, vision-driven UI automation across multiple platforms, enabling natural-language control for developers and power users. It provides reliable cross-platform support by operating solely from screenshots, abstracting away platform-specific complexities and simplifying UI automation tasks.

How It Works

The core approach leverages Midscene.js's vision-based automation capabilities. By processing screenshots, it achieves robust cross-platform compatibility without relying on traditional DOM or element inspection. This visual grounding allows natural language commands to be translated into precise UI interactions, offering a novel way to automate user interfaces.

Quick Start & Requirements

Installation is straightforward using Node.js and the command npx skills add web-infra-dev/midscene-skills. A critical prerequisite is setting up access to suitable AI models with strong visual grounding. This involves configuring environment variables such as MIDSCENE_MODEL_API_KEY, MIDSCENE_MODEL_NAME, MIDSCENE_MODEL_BASE_URL, and MIDSCENE_MODEL_FAMILY, with examples provided for models like Gemini, Qwen3-VL, and Doubao Seed 1.6. Further details are available at https://midscenejs.com/model-strategy and https://midscenejs.com/model-common-config.

Highlighted Details

• Cross-Platform Support: Includes skills for Browser (Puppeteer), Chrome Bridge, Desktop (macOS, Windows, Linux), Android (ADB), iOS (WebDriverAgent), and HarmonyOS (HDC).
• E2E Testing Integration: Features Vitest integration for scaffolding, converting, and managing AI-powered End-to-End tests.
• Natural Language Control: Enables users to interact with UIs using plain language commands.

Maintenance & Community

Bug reports and feature requests should be directed to the main web-infra-dev/midscene repository's issues page. Specific community channels like Discord or Slack are not detailed in this README.

Licensing & Compatibility

The project is released under the permissive MIT license, generally allowing for commercial use and integration into closed-source projects without significant restrictions.

Limitations & Caveats

A safety warning highlights that AI-driven UI automation may yield unpredictable results due to its ability to control all screen elements. Users are advised to carefully evaluate the associated risks before deployment.