Summary

Cola DLM introduces a hierarchical latent-variable language model that decouples global semantic organization from local textual realization. It targets researchers and developers seeking advanced text generation capabilities, offering a HuggingFace-native, efficient inference pipeline and an OpenAI-compatible API for seamless integration and deployment.

How It Works

The model employs a two-stage approach: a Text VAE maps text to a stable continuous latent space, and a block-causal Diffusion Transformer (DiT) models the latent prior using Flow Matching. This architecture decomposes the latent space into blocks, enabling a block-causal factorization. Inference involves encoding a prefix, performing block-wise latent prior transport, and then decoding tokens, facilitating generation without fixed-length padding.

Quick Start & Requirements

• Installation: Clone the repository and install via pip install -e . or pip install -e ".[dev]".
• Prerequisites: Python 3.9+, PyTorch 2.1+, HuggingFace Transformers 4.40+. Operates on Linux/macOS.
• Resources: Model weights are available on HuggingFace (ByteDance-Seed/Cola-DLM).
• Documentation: Project Page: https://hongcanguo.github.io/Cola-DLM/. Paper: https://arxiv.org/abs/2605.06548.

Highlighted Details

• HuggingFace-Native: Models (ColaDiTModel, ColaTextVAEModel) subclass transformers.PreTrainedModel for seamless integration with the HuggingFace ecosystem.
• No-Padding Inference: Implements a "NA" (no-padding) flatten-concat pipeline for efficient, variable-length sample generation.
• Block-Causal Prior: Utilizes a block-causal DiT and classifier-free guidance for structured latent generation.
• OpenAI-Compatible Serving: An openai_adapter/ module exposes a /v1/chat/completions API endpoint.
• Reproducible Benchmarks: Includes scripts (scripts/run_benchmark.sh) to reproduce the paper's 8-task evaluation.

Maintenance & Community

The project is authored by ByteDance Seed et al. The README details the research team and paper authors but does not specify ongoing maintenance plans, community channels (e.g., Discord, Slack), or a public roadmap.

Licensing & Compatibility

Released under the Apache License 2.0. This permissive license allows for commercial use and integration into closed-source projects.

Limitations & Caveats

Unified text-image modeling experiments are preliminary and not included in this release. Minor fluctuations in per-task accuracy may be observed compared to the paper's internal implementation due to differences in the HuggingFace Transformers-based code.