This JavaScript class enables real-time lip-sync for 3D avatars, targeting web developers and creators who want to build interactive experiences with talking characters. It leverages Ready Player Me avatars and Three.js for rendering, offering dynamic lip-sync, facial expressions, and avatar animations.

How It Works

The TalkingHead class integrates with TTS services to generate visemes (facial movements) synchronized with spoken audio. It uses Three.js for 3D rendering and supports custom Ready Player Me avatars and Mixamo animations. The core functionality relies on mapping audio timestamps to viseme data, allowing for real-time facial animation. It also includes features for dynamic bones, custom poses, moods, and gestures, enhancing avatar expressiveness.

Quick Start & Requirements

• Install: Import from CDN or download modules.
• Prerequisites: Modern web browser (Chrome, Firefox, Safari, Edge), Three.js/WebGL.
• Setup: Minimal setup for basic use; integration with TTS/AI services requires API keys and potentially proxy configurations.
• Docs: https://github.com/met4citizen/TalkingHead#readme

Highlighted Details

• Real-time lip-sync with support for multiple languages (English, Finnish, Lithuanian beta).
• Integration with various TTS and AI services (Google TTS, Microsoft Azure, OpenAI, ElevenLabs).
• Dynamic bone physics for realistic hair and cloth movement.
• Extensive customization options for avatar appearance, animations, poses, and moods.
• Streaming API for low-latency audio and lip-sync updates.

Maintenance & Community

The project appears to be a personal side-project with no explicit mention of active maintenance or community channels like Discord/Slack.

Licensing & Compatibility

• TalkingHead Class: No explicit license mentioned in the README.
• Example Assets: Ready Player Me avatars (CC BY-NC 4.0 for non-commercial use), Mixamo animations (royalty-free for personal, commercial, non-profit projects, with restrictions on distribution and ML training).

Limitations & Caveats

• The project is described as a "small side-project" with no major future plans.
• Custom avatar creation requires specific rigging and blendshapes compatible with Ready Player Me standards.
• Integration with paid TTS services requires API keys and potentially complex proxy setups for secure authentication.
• Lip-sync accuracy for languages other than English and Finnish may vary, requiring custom module development.