This project provides Unity developers with sample code and helper classes to access and utilize the passthrough camera feed on Meta Quest devices. It targets developers building mixed reality experiences who need to integrate real-world camera data into their Unity applications, offering functionalities like displaying the feed, performing computer vision tasks, and applying visual effects.

How It Works

The project leverages Unity's WebCamTexture class to access the Quest's passthrough camera, abstracting the underlying Android Camera2 API. It includes manager classes for handling permissions, initialization, and retrieving camera metadata. Sample scenes demonstrate converting 2D image coordinates to 3D world space using camera intrinsics and poses, integrating with Unity Sentis for on-device ML/CV inference, and applying custom GPU shaders to the camera feed.

Quick Start & Requirements

• Install: Clone the repository using git clone https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples. Ensure Git LFS is installed (git lfs install).
• Requirements:
  • Unity 2022.3.58f1 or 6000.0.38f1.
  • Meta MRUK (v74.0.0+).
  • Unity Sentis (v2.1.1).
  • Meta Quest Devices (Quest 3/3S) running Horizon OS v74+.
  • Android permissions: android.permission.CAMERA, horizonos.permission.HEADSET_CAMERA.
• Setup: Open the project in Unity and use the "Meta > Tools > Project Setup Tool" to fix configuration issues.
• Docs: Meta Developers Documentation

Highlighted Details

• Demonstrates 2D to 3D coordinate conversion for precise object placement using raycasting with MR Utility Kit.
• Includes an object detection sample using Unity Sentis with a YOLO model optimized for Quest 3.
• Offers a shader sample for applying custom GPU effects directly to the passthrough feed.
• Provides helper classes for simplified permission handling and camera data access.

Maintenance & Community

The project is maintained by Meta. Further community interaction details are not explicitly provided in the README.

Licensing & Compatibility

The Oculus License applies to the SDK and supporting materials. The MIT License applies to specific, marked files (e.g., models in SentisInference/Model). Commercial use is generally permitted under the Oculus License, but specific restrictions may apply.

Limitations & Caveats

The WebCamTexture implementation has limitations, including single camera access at a time and a slight delay (40-60ms) due to missing timestamps. The maximum supported resolution is 1280x960, with potential cropping. Concurrent permission management with other systems may cause conflicts. The object detection model has known accuracy limitations and class granularity issues. Meta XR Simulator and Meta Link are not supported.