SAGE: Scalable Agentic 3D Scene Generation for Embodied AI

This repository provides SAGE, an agentic framework for automatically generating simulation-ready 3D environments tailored to specific embodied AI tasks. It targets researchers and engineers in embodied AI and robotics, offering a scalable pipeline to create diverse, interactive 3D scenes and large-scale datasets, thereby accelerating development and testing.

How It Works

SAGE employs an agentic approach, leveraging Foundation Models (LLMs and VLMs) to interpret user-defined task intents. The framework orchestrates the generation of realistic 3D scenes, including layouts, objects, and materials, integrating with NVIDIA Isaac Sim for simulation. This automated, task-driven pipeline enables the scalable creation of complex virtual environments.

Quick Start & Requirements

Setup requires configuring both the server (backend) and client (frontend) components. Key dependencies include Foundation Models (e.g., Qwen, GPT), 3D generation models (TRELLIS), NVIDIA Isaac Sim, and integrations with Isaac Lab, M2T2, MatFuse, and RoboMimic. Detailed setup instructions are available in the respective Server and Client documentation.

Highlighted Details

• SAGE-10k Dataset: A large-scale dataset featuring 10,000 diverse indoor scenes across 50 room types/styles, populated with 565K unique 3D objects.
• Agentic Generation: Automatically generates simulation-ready environments based on user-specified embodied tasks.
• Integrated Toolchain: Incorporates TRELLIS for 3D asset generation, MatFuse for material synthesis, M2T2 for manipulation data, and RoboMimic for policy learning.

Maintenance & Community

The repository lists multiple authors from NVIDIA, University of Illinois Urbana-Champaign, Cornell University, and Stanford University. Specific community channels (e.g., Discord, Slack), roadmaps, or dedicated maintenance teams are not detailed in the provided README.

Licensing & Compatibility

The core SAGE framework's license is not explicitly stated. However, integrated components have varying licenses: IsaacLab (BSD-3-Clause), M2T2 (NVIDIA License), MatFuse-SD (MIT), RoboMimic (MIT), TRELLIS (MIT), and others like Holodeck, HoloScene, DRAWER, Flux, Qwen3 (Apache 2.0). Modifications to IsaacLab, M2T2, MatFuse-SD, and RoboMimic are subject to their original licenses, potentially introducing complexity for commercial use or integration.

Limitations & Caveats

The README does not explicitly list limitations. However, the reliance on modified versions of several external repositories (IsaacLab, M2T2, MatFuse, RoboMimic) suggests potential challenges in dependency management, version compatibility, and setup complexity. The specific terms of the NVIDIA License for M2T2 may impose usage restrictions.