LoongForge is a unified, high-performance training framework designed for large-scale AI models, including LLMs, VLMs, diffusion, and embodied AI. It targets researchers and engineers seeking to accelerate training across diverse architectures and hardware, offering significant speedups and a modular, scalable approach.

How It Works

Built upon Megatron-LM, LoongForge introduces deep systemic enhancements for improved training performance and broader hardware support. Its core design emphasizes modularity and scalability, featuring heterogeneous parallelism for independent component tuning, decoupled encoder-decoder training to eliminate pipeline bubbles, MoE-native optimizations, and adaptive FP8 precision for efficiency. Custom fused operators and versatile data pipelines further streamline the training workflow.

Quick Start & Requirements

Installation is supported via source build or Docker (prebuilt images are forthcoming). The framework requires NVIDIA GPUs or Kunlun XPUs, with specific installation guides available for each. Users can explore ready-to-run scripts in the configs/, examples/, and examples_xpu/ directories. Official documentation is available at https://loongforge.readthedocs.io/en/latest/index.html.

Highlighted Details

• Achieves up to 5.04x training speedup on DeepSeek-V3.2 Lite compared to baseline Megatron-LM, leveraging DSA CUDA kernel optimizations.
• Provides native support for both NVIDIA GPUs and Kunlun XPUs through a plugin design.
• Features flexible multi-modal composition and heterogeneous parallelism for optimizing throughput and memory across model components.
• Includes MoE-native optimizations and adaptive FP8 training for enhanced LLM and VLM training efficiency.

Maintenance & Community

LoongForge is actively developed, with its first official release (v0.1.0) in May 2026. Community engagement is facilitated via Slack and WeChat channels. Contributions are welcomed following the project's contributing guidelines.

Licensing & Compatibility

The project is released under the permissive Apache License 2.0, allowing for broad compatibility with commercial and closed-source applications.

Limitations & Caveats

Prebuilt Docker images are not yet available. Benchmarking for certain models, such as DeepSeek-V3.2, was conducted on reduced-layer configurations due to hardware limitations, suggesting potential performance variations at full scale.