Summary

OpenMythos offers a theoretical, open-source reconstruction of the Claude Mythos architecture, implementing a Recurrent-Depth Transformer (RDT). It targets researchers and engineers exploring advanced LLM designs, providing capabilities for compute-adaptive reasoning, depth extrapolation, and parameter-efficient deep reasoning through its unique looped architecture.

How It Works

The core is a Recurrent-Depth Transformer (RDT) with Prelude, looped Recurrent Block, and Coda stages. It features switchable Multi-Head Attention (MLA) or Grouped-Query Attention (GQA) and a sparse Mixture of Experts (MoE) feed-forward network with routed and shared experts. Training stability is ensured by constraining injection parameters to maintain a spectral radius less than 1, preventing divergence.

Quick Start & Requirements

Install via pip install open-mythos. The project requires Python and PyTorch. Training scripts support multi-GPU setups (torchrun) and specify precision (bfloat16/float16). Pre-configured model variants range from 1B to 1T parameters. API documentation is available at docs/open_mythos.md.

Highlighted Details

• Parameter Efficiency: Achieves high-quality deep reasoning with fewer parameters by recycling weights in the recurrent block.
• Systematic Generalization & Depth Extrapolation: The RDT architecture enables handling novel compositions and extending reasoning chains via increased inference-time loops.
• Implicit Chain-of-Thought: Reasoning occurs in continuous latent space across loop iterations, simulating CoT steps and enabling breadth-first reasoning exploration.
• Mixture of Experts (MoE): Fine-grained and shared experts allow the model to handle diverse domains, with routing potentially adapting across loop iterations.

Maintenance & Community

This project is community-driven and theoretical. Specific details regarding active maintenance, contributor recognition, or dedicated community channels are not provided in the README.

Licensing & Compatibility

Released under the permissive MIT License, allowing for broad compatibility with commercial and closed-source applications.

Limitations & Caveats

As a theoretical reconstruction, OpenMythos is not an official Anthropic implementation. The architecture is biased towards compositional reasoning, potentially leading to inconsistent factual recall. An Adaptive Computation Time (ACT) mechanism is likely used to mitigate "overthinking," but its implementation details are not fully elaborated.