LegoGPT generates physically stable and buildable toy brick models from text prompts, targeting researchers and developers interested in generative design and physical simulation. It enables the creation of tangible, structurally sound designs from natural language descriptions.

How It Works

LegoGPT leverages a fine-tuned Llama-3.2-1B-Instruct model to translate text prompts into LEGO brick configurations. It incorporates a physics-based stability analysis using Gurobi (or a simpler connectivity check) to ensure generated models are physically sound. The output includes a rendered image, a text-based brick list, and an LDraw file for visualization and compatibility.

Quick Start & Requirements

• Install: Clone the repo and use uv sync for dependency management.
• Prerequisites:
  • Hugging Face user access token (HF_TOKEN environment variable) for Llama-3.2-1B-Instruct.
  • Gurobi license (optional, academic licenses available) for physics-based stability analysis.
  • LDraw parts library downloaded and configured via LDRAW_LIBRARY_PATH.
• Demo: Interactive inference via uv run infer.
• Docs: Installation and usage details are within the README.

Highlighted Details

• Generates physically stable LEGO structures from text prompts.
• Outputs include rendered images, text descriptions, and LDraw files.
• Stability analysis can be physics-based (Gurobi) or connectivity-based.
• Fine-tuning scripts and dataset preparation instructions are provided.

Maintenance & Community

The project is associated with Carnegie Mellon University and acknowledges several contributors and funding sources, including Packard Foundation, Cisco, Amazon, and Microsoft Research. The codebase builds upon Hugging Face TRL and Accelerate.

Licensing & Compatibility

The core LegoGPT model, dataset, and most code are licensed under the MIT License. Submodules ImportLDraw and FlashTex have their own licenses, which should be reviewed for specific usage terms. The MIT license generally permits commercial use and linking with closed-source projects.

Limitations & Caveats

Requires access to a gated Llama model. Gurobi, while recommended for accuracy, requires a separate license. The stability analysis is a key component, and its accuracy depends on the chosen method and Gurobi availability.