SightFlow is an open-source desktop runtime engine designed to bridge the gap between Large Language Models (LLMs) and real-world software interactions. It enables AI agents to "see" any graphical user interface (GUI), understand its state, plan actions, execute them like a human operator, and learn from each execution. This empowers AI to perform complex, multi-step tasks across diverse desktop applications, even those lacking APIs, by interpreting screen pixels into structured semantics. The primary benefit is enabling AI to learn jobs and operate effectively in real-world workflows, moving beyond LLMs' capabilities in thinking and speaking to include doing and learning.

How It Works

SightFlow operates on a "See · Think · Do · Learn" paradigm. A vision model (SEE) interprets any GUI and its current state. The agent then plans actions based on context and history (THINK). It executes these actions—clicking, typing, switching windows—like a human operator (DO). Crucially, every execution is recorded as a structured "work-trace," capturing the UI state, rationale, action, and result (LEARN). This work-trace forms a durable working memory, offering capabilities beyond traditional RPA by recording why each step was taken, enabling replay, evaluation, and inheritance of judgment.

Quick Start & Requirements

• Primary install / run command: Use npm install to install dependencies, npm run dev to run in development, and npm run build:<platform> (e.g., build:win) for release builds.
• Non-default prerequisites and dependencies: Node.js (LTS) and npm are required. A Volcengine Ark API key is necessary for initial setup, visual grounding, and the built-in Doubao agent.
• Links: Website: https://sightflow.dev. A Discord server is available for community engagement.

Highlighted Details

• Universal Vision-Based RPA: Interacts with any desktop software by interpreting visual cues, eliminating the need for fragile webhooks or private protocols.
• State-of-the-Art Vision: A unified vision layer extracts real-time UI state, message lists, and chat-bubble text from complex, dynamic layouts.
• Agentic Workspaces: Transforms unstructured chat requests into actionable node-workflows and API calls, fully programmable via local AI.
• Local-First Execution & Data Ownership: Work traces are stored locally by default, ensuring data privacy and that user data never leaves the machine.
• Fully Auditable: Every action and decision is captured in a structured work-trace, allowing for end-to-end inspection and step-by-step replay.
• Model-Agnostic: Designed to adapt to and switch between different vision LLMs and providers.

Maintenance & Community

The project encourages community involvement, with an active Discord server for co-building and a welcome for issues and pull requests.

Licensing & Compatibility

Released under the Apache License 2.0, this permissive license generally allows for commercial use and integration with closed-source projects.

Limitations & Caveats

Initial setup requires obtaining and configuring a Volcengine Ark API key, which may involve external service costs or account setup. While model-agnostic, users must select and configure an agent/provider for specific tasks, though a default is included. The project appears to be in active development, positioning itself as a foundational "working memory engine."