This project demonstrates voice-based interaction with a 3D solar system model using OpenAI's Realtime API and WebRTC. It targets developers and users interested in integrating natural language control with interactive 3D environments, enabling dynamic scene manipulation and data visualization through conversational commands.

How It Works

The demo leverages OpenAI's Function Calling mechanism to interpret user voice commands. These commands are mapped to specific tools that trigger actions within a Spline-rendered 3D solar system. The Realtime API facilitates low-latency communication, enabling immediate visual feedback and scene updates based on spoken queries about planets, their data, or celestial events like the ISS's position.

Quick Start & Requirements

• Install dependencies: npm install
• Run the app: npm run dev
• Requires an OpenAI API key, set via the OPENAI_API_KEY environment variable or a .env file.
• Official documentation: https://github.com/openai/openai-realtime-solar-system

Highlighted Details

• Voice commands trigger animations for specific planets and moons within the 3D scene.
• Data-driven queries generate dynamic charts (bar or pie) within the UI.
• Real-time ISS position fetching and visualization in the 3D environment.
• Camera controls and scene resets are also voice-activated.

Maintenance & Community

This is an official OpenAI demo project. Further community engagement or maintenance details are not specified in the README.

Licensing & Compatibility

• Licensed under the MIT License.
• Permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

The initial scene load can be slow due to its size. Voice recognition may be sensitive to background noise or echo. Customization requires familiarity with Spline's event system and modifying TypeScript configuration files.