TipTour offers an open-source, AI-powered macOS companion for voice and screen control. It enables users to point, click, type, open apps, edit text, or act on highlighted screen areas via natural language, enhancing productivity.

How It Works

TipTour integrates Gemini Live for real-time voice, screen understanding, and tool calling. It uses CUA Driver Core for computer control (clicks, typing) and macOS Accessibility for native app structure analysis. A unique "Focus Highlight" feature lets users paint a freeform screen area (Ctrl+Shift+drag) for context-specific commands. This enables precise, AI-driven computer manipulation.

Quick Start & Requirements

Building from source requires macOS 14+, Xcode 16+, and Node 20+ (for optional Cloudflare Worker). Setup involves opening tiptour-macos.xcodeproj in Xcode, selecting a signing team, building (Cmd+R), pasting a Gemini API key into the menu bar panel, and granting macOS permissions (Microphone, Screen Recording, Accessibility, Screen Content). The API key is stored in macOS Keychain.

Highlighted Details

• Focus Highlight: Precise, freeform screen region selection for targeted commands.
• Contextual Actions: Commands operate within the active app and selected/hovered elements.
• Operating Modes: Includes Autopilot (automatic), Tour Guide (tutorials), and Neko Mode (playful cursor).
• Privacy-Focused: Outlines permissions and secures Gemini API key locally.

Maintenance & Community

The provided README does not contain specific details regarding notable contributors, sponsorships, community channels (like Discord or Slack), roadmaps, or deprecation notices.

Licensing & Compatibility

The project is licensed under the MIT license, which generally permits commercial use and integration into closed-source projects. Compatibility is restricted to macOS 14 and later.

Limitations & Caveats

TipTour is macOS-only (14+). Users must supply their own Gemini API key, incurring potential costs and privacy concerns. Building requires specific Xcode/Node.js versions. Potential TCC permission state invalidation exists when using xcodebuild for testing.