Summary

Molt is an agentic-first, PyTorch-native framework for multimodal, multi-turn agentic Reinforcement Learning (RL) research. It offers a fully-async stack using Ray and vLLM, enabling rapid iteration on agent environments and scaling training to 1T-class Mixture-of-Experts (MoE) models. Its primary benefit is a minimal, hackable codebase that simplifies complex RL setups for frontier-scale research.

How It Works

Molt adopts an "agent is the program" philosophy, abstracting RL infrastructure. It leverages Ray for distributed execution and async queues, vLLM for rollout inference, and NVIDIA AutoModel with FSDP2 for PyTorch-native training. A token-first contract ensures alignment of token IDs, logprobs, rewards, and multimodal tensors. This design allows any Python computation, including tool use or LLM-as-judge, to serve as a reward signal, fostering rapid agent experimentation.

Quick Start & Requirements

The recommended installation is via a pre-built Docker image (hijkzzz/molt:latest) containing the full CUDA stack. Local development uses pip install -e ".[vllm]". Effective use, especially for large models, requires NVIDIA GPUs (e.g., A100/H100/H200/B200/GB200) and Python 3.10+. Quick-start scripts for SFT and RL are available in the repository.

Highlighted Details

• Scalability: Supports 1T-class MoE models via NVIDIA AutoModel with TP, EP, CP parallelism, FSDP2, and optional Adam CPU offload.
• Agent Interface: A unified Gymnasium-aligned API (Env.step() or ChatAgent.run()) handles graders, multi-turn tools, VLM environments, and OpenAI/Anthropic servers.
• Codebase: Features ~8.6K lines of RL code, designed for readability and customizability.
• Advanced RL: Integrates Importance Sampling (IS) correction, MoE routing stability (Router Replay, Router Freeze), and on-policy distillation.

Maintenance & Community

Maintained by "Molt Contributors." External contributions are welcomed via CONTRIBUTING.md. No specific community channels (e.g., Discord, Slack) are detailed.

Licensing & Compatibility

Released under the Apache License 2.0, which permits commercial use and integration into closed-source projects.

Limitations & Caveats

The muon optimizer is experimental. Context Parallelism (CP) training currently disables packed RL batches. Router Replay is incompatible with partial rollout enablement. MTP rollout support depends on vLLM and model checkpoint specifics.