This repository provides a collection of Unity projects demonstrating advanced AR/VR applications using Meta Quest's Passthrough Camera API (PCA). It targets XR developers looking to leverage real-world environmental data for enhanced tracking, visual effects, and AI integration. The benefit is a practical toolkit for building sophisticated mixed-reality experiences.

How It Works

The project utilizes Unity's WebcamTexture to access the Quest's passthrough cameras, enabling real-time environmental data capture. It showcases various techniques, including mapping 3D space to 2D pixels for color picking, integrating Unity Sentis for on-device object detection, and using ZXing for QR code tracking. Advanced samples explore OpenAI vision models for scene understanding and WebRTC for streaming passthrough video.

Quick Start & Requirements

• Install: Clone the repository and open in Unity.
• Prerequisites: Meta Quest 3 or 3s with HorizonOS v74+, Unity 6 (or 2022.3 LTS). PCA does not work in the Editor or XR Simulator.
• Dependencies: Unity Sentis (for Object Detection), ZXing.Net (for QR Code Tracking), NativeWebSocket and SimpleWebRTC (for WebRTC streaming).
• Setup: Requires building and running APKs on the headset.
• Docs: Meta Quest Developer Documentation

Highlighted Details

• Object Detection sample runs at ~60 fps using Unity Sentis.
• QR Code Tracking achieves ~70 fps for multiple codes and 72 fps for single codes.
• OpenAI vision model integration takes 2-6 seconds per loop.
• WebRTC streaming requires a separate WebSocket signaling server.

Maintenance & Community

The project is actively maintained by xrdevrob and features contributions from various developers. A Discord community is available for discussion and support.

Licensing & Compatibility

Licensed under the MIT License, allowing for commercial use and integration into closed-source projects. Attribution is appreciated.

Limitations & Caveats

PCA is currently experimental and apps using it cannot be submitted to the Meta Horizon Store. The WebcamTexture API supports only one camera (left or right) at a time. WebRTC streaming requires specific configuration for graphics API (OpenGLES3) and disabling certain XR Plug-in Management settings.