This project enables users to create augmented reality experiences using natural language commands, powered by ChatGPT. It's targeted at developers and researchers interested in AI-driven AR content creation, offering a novel approach to generating AR scenes through spoken prompts and custom APIs.

How It Works

ChatARKit leverages a custom JavaScript environment within an iOS app, powered by JavaScriptCore. Spoken commands are transcribed using Whisper, then sent to ChatGPT with a preamble describing available custom JavaScript functions (e.g., createEntity, getPlanes). ChatGPT generates JavaScript code to manipulate ARKit entities, which is then executed within the app. This approach allows for dynamic AR scene generation based on natural language, with Sketchfab integration for 3D assets.

Quick Start & Requirements

• Install: Open iOS/ChatARKit/ChatARKit.xcodeproj and deploy to an iPhone.
• Prerequisites:
  • Whisper model weights (download via curl command provided).
  • OpenAI API Key (paste into APIKeys.swift).
  • Sketchfab API Token (paste into APIKeys.swift).
  • Xcode for deployment.
• Setup Time: Estimated to be under 30 minutes, assuming API keys are readily available.
• Links: Demo Video

Highlighted Details

• Integrates ChatGPT for natural language to AR code generation.
• Uses Whisper for speech-to-text transcription.
• Leverages JavaScriptCore for executing generated AR manipulation code.
• Imports 3D models from Sketchfab based on textual descriptions.

Maintenance & Community

The project is a personal experiment by Bart Trzynadlowski. Further development and robustness improvements are encouraged via direct contact.

Licensing & Compatibility

The project is Copyright 2022-2023 Bart Trzynadlowski. Specific licensing details are not provided in the README, but it relies on other projects with their own licenses (whisper.cpp, GLTFSceneKit, Zip, pyChatGPT). Commercial use compatibility is not explicitly stated.

Limitations & Caveats

Performance is variable, with ChatGPT frequently generating incorrect code or referencing non-existent functions. Sketchfab model downloads and imports can fail, with no on-screen status updates. The project is described as an experiment, and users should examine generated code and monitor Xcode console logs for debugging.