This repository provides the official implementation for Engram, a novel conditional memory module designed to enhance Large Language Models (LLMs) by introducing a new axis of sparsity. It addresses the inherent lack of native knowledge lookup primitives in Transformers, complementing existing Mixture-of-Experts (MoE) approaches. Engram offers a scalable lookup mechanism, enabling improved model capacity and performance under strict parameter and FLOP constraints, particularly benefiting knowledge-intensive tasks and complex reasoning.

How It Works

Engram augments Transformer backbones by retrieving static N-gram memory and fusing it with dynamic hidden states. It modernizes classic N-gram embeddings for efficient O(1) lookup. This approach is advantageous as it formulates a trade-off between neural computation (MoE) and static memory, guided by a U-shaped scaling law for optimal capacity allocation. Deterministic addressing allows massive embedding tables to be offloaded to host memory with minimal inference overhead, enhancing system efficiency.

Quick Start & Requirements

• Installation: pip install torch numpy transformers sympy
• Prerequisites: Python 3.8+, PyTorch.
• Demo: A standalone implementation engram_demo_v1.py is provided to illustrate the core logic, mocking standard components.
• Note: The demo code is intended for illustrating data flow and the Engram module's functionality.

Highlighted Details

• The Engram-27B model shows consistent improvements over MoE baselines across knowledge, reasoning, code, and math domains under iso-parameter and iso-FLOPs constraints.
• Mechanistic analysis suggests Engram relieves early layers from static pattern reconstruction, potentially preserving effective depth for complex reasoning.
• The module employs deterministic addressing for efficient offloading of massive embedding tables to host memory.

Maintenance & Community

• Contact: For questions, raise an issue on the repository or contact service@deepseek.com.

Licensing & Compatibility

• License: Use of Engram models is subject to a specific "Model License" (details not provided in the README).
• Compatibility: No explicit notes on commercial use or closed-source linking are present; the "Model License" should be consulted.

Limitations & Caveats

• The provided code is a demonstration version, mocking standard components to highlight the Engram module's data flow.
• The specific terms and restrictions of the "Model License" are not detailed in the README, requiring further investigation for adoption.