Summary

GRAIL is a fully digital pipeline for synthesizing humanoid loco-manipulation data. It generates metric 4D human-object interaction (HOI) trajectories from 3D assets and video priors, retargets them to a Unitree G1 robot, and trains task-general policies. This enables robust sim-to-real transfer for complex robotic tasks using purely synthetic data.

How It Works

The system composes 3D assets, simulator scenes, robot models, and video foundation models to synthesize 4D HOI trajectories. These trajectories are then retargeted to a Unitree G1 robot, facilitating the training of task-general policies for manipulation and locomotion. This approach leverages a fully digital workflow for large-scale, diverse data generation, crucial for sim-to-real transfer.

Quick Start & Requirements

• Install: Primarily via Docker. Inside the container, run scripts/setup/install_env_docker.sh (installs Blender) and scripts/setup/download_checkpoints.sh (GEM-SMPL, GEM-SOMA, FoundationPose weights). Activate environment with conda activate grail and source .env.
• Prerequisites: Docker, NVIDIA GPUs (--gpus all), shm_size=16g, Conda, Blender, specific model checkpoints, and API keys (OPENAI_API_KEY, KLING_*, HF_TOKEN).
• Documentation: Full documentation available at docs and markdown sources.

Highlighted Details

• Synthesizes metric 4D HOI trajectories for human-object interactions.
• Trains task-general policies for pick-up, manipulation, sitting, and terrain traversal (stairs, curb, slope).
• Demonstrates sim-to-real transfer to a real-world Unitree G1 robot for object pick-up and stair-climbing, using exclusively GRAIL-generated data.
• Integrates Kling video foundation model priors for data generation.

Maintenance & Community

The README does not detail specific contributors, sponsorships, community channels (e.g., Discord/Slack), or a public roadmap. TODOs indicate planned releases for the GRAIL manipulation dataset and task-general tracking policy checkpoints.

Licensing & Compatibility

Released under the NVIDIA License. The Work and derivative works are restricted to non-commercial use, except for NVIDIA Corporation and its affiliates. This license significantly limits commercial adoption or integration into proprietary systems.

Limitations & Caveats

The primary limitation is the non-commercial use restriction. Additionally, the GRAIL manipulation dataset and task-general tracking policy checkpoints are not yet released, as indicated by project TODOs.