This open-source Android application provides real-time screen translation for games, visual novels, and manga without requiring root access. It supports on-device and cloud OCR, offline LLMs, multiple translation services, and displays translations directly on screen, enhancing accessibility for multilingual content consumption.

How It Works

The project employs a modular pipeline: screen capture (MediaProjection/Shizuku) feeds into OCR (on-device or cloud) and translation engines (LLM/MT). Results are overlaid dynamically. This approach offers flexibility, allowing users to mix and match components, and enables full offline operation or self-hosted services for privacy.

Quick Start & Requirements

• Install: Download the latest ScreenTranslator-x.y.z.apk from releases.
• Permissions: Requires Overlay, Notification, and optionally Accessibility (for volume key trigger) or Shizuku (for persistent screen capture).
• Architecture: Primarily arm64-v8a. 32-bit ARM and x86 are unsupported.
• Dependencies: On-device OCR/LLM models require significant storage and Wi-Fi for download.

Highlighted Details

• Flexible Pipeline: Users can combine various on-device/cloud OCR (ML Kit, PaddleOCR, Manga OCR) and translation services (OpenAI-compatible LLMs, DeepL, Google, Sakura/Hy-MT2 offline LLMs).
• Offline & Self-Hosted: Full offline functionality is achievable with on-device models. Users can also self-host OCR (Umi-OCR) and translation (DeepLX) services.
• Contextual Translation: A term library maintains consistent translations for names/jargon, with LLM support for app-specific context.
• Advanced Display: Offers "BLOCKS" mode (near original text) or a floating window. Features automatic screen adaptation, customizable styles, and optimized rendering for vertical text and long translations.
• Word-by-Word Dictionary: LLM integration provides detailed dictionary entries for selected words/phrases.

Maintenance & Community

• Community: QQ Group 1059655926 available.
• Development: Follows Conventional Commits; PRs target the dev branch after issue discussion. main is reserved for releases.

Licensing & Compatibility

• License: Apache-2.0 for the code, allowing commercial use and closed-source linking. Models retain their original licenses.

Limitations & Caveats

• Android 14+ Authorization: Persistent screen capture requires re-authorization on first launch unless Shizuku is used.
• ROM Issues: Some Android ROMs aggressively manage background processes, potentially interfering with overlays.
• Security/Compatibility: May be flagged by anti-cheat systems; does not bypass FLAG_SECURE screens.
• Performance: On-device models can be slow on lower-end devices; large model downloads require Wi-Fi. Experimental features (e.g., Vulkan acceleration) are off by default.
• Network Access: Some model sources (Hugging Face) may require proxies.