LERF (Language Embedded Radiance Fields) enables users to interactively query and manipulate 3D scenes represented by Neural Radiance Fields (NeRFs) using natural language. It targets researchers and developers in computer vision and graphics who want to explore semantic control over 3D scene generation and editing. The primary benefit is enabling intuitive, text-based scene exploration and modification.

How It Works

LERF integrates with the Nerfstudio framework, extending its NeRF models with a language field. It leverages pre-trained vision-language models (like CLIP or DINO) to embed textual descriptions into the 3D scene representation. This allows for semantic querying, where specific regions or objects in the NeRF can be identified and visualized based on text prompts, facilitating a new paradigm for 3D scene interaction.

Quick Start & Requirements

• Install: Follow Nerfstudio's custom method integration guidelines, including tinycudann. Clone the repo (git clone https://github.com/kerrj/lerf) and install as a package (python -m pip install -e .). Verify with ns-train -h.
• Prerequisites: CUDA (implied by Nerfstudio), Python. GPU memory is a key consideration.
• Usage: Launch training with ns-train lerf --data <data_folder>. Connect to the viewer via the provided link.
• Docs: Nerfstudio documentation

Highlighted Details

• Supports multiple vision-language encoders (CLIP, OpenCLIP).
• Offers lerf-lite for reduced memory footprint and lerf-big with ViT-L/14 for larger models.
• Enables visualization of "relevancy maps" based on text queries.

Maintenance & Community

• Issues can be reported via GitHub issues.
• The project is associated with the ICCV 2023 paper.

Licensing & Compatibility

• The repository does not explicitly state a license in the provided README.

Limitations & Caveats

The project has a "TODO" for integrating command-line video rendering with custom prompts. Visualization code may change as Nerfstudio features evolve, requiring users to check for updates when forking. High-resolution rendering (above 256px) may be slow.