Summary

TinyEngram explores the Engram architecture for efficient LLM fine-tuning and memory injection, targeting researchers and developers. It offers parameter-efficient methods outperforming LoRA in catastrophic forgetting resistance and extends to vision models like Stable Diffusion for lightweight, composable concept injection, enhancing model adaptability and specialized knowledge integration.

How It Works

TinyEngram integrates a compact N-gram memory module and gated retrieval into transformer layers for enhanced phrase-level understanding and concept injection. For vision, it injects learned embeddings into Stable Diffusion's Text Encoder via prompt N-grams, freezing the backbone. This enables lightweight, non-interfering concept additions, treating visual concepts as retrievable "memories" through exact N-gram matching.

Quick Start & Requirements

• Installation: Requires Python 3.10. Setup involves creating/activating a Conda environment (conda create -n tinyengram python=3.10 -y; conda activate tinyengram), upgrading pip (pip install --upgrade pip), and installing dependencies (pip install -r requirements.txt). CUDA notes are in doc/reproduction/environment.md.
• Resources: Technical reports are available via ./doc/paper/tinyengram_vision_paper.pdf or arXiv. Reproduction guides and issue reporting are linked within the repository.

Highlighted Details

• Demonstrates superior resistance to catastrophic forgetting compared to LoRA, preserving general capabilities during adaptation.
• Successfully extends to vision models (Stable Diffusion) for lightweight, composable concept injection without fine-tuning the core backbone.
• Provides parameter-efficient fine-tuning, achieving improved performance on specialized tasks with minimal added parameters.
• Injected memories are infinitely composable due to exact N-gram matching, preventing interference.

Maintenance & Community

The project promotes open research, encouraging community input via GitHub Issues for experiments and questions. All code, logs, and experiments are openly shared. Specific community channels or contributor/sponsorship details are not provided.

Licensing & Compatibility

The README does not specify a software license, requiring clarification for commercial use or closed-source integration.

Limitations & Caveats

A trade-off exists in Engram's vocabulary scalability: smaller capacities risk semantic collisions, while larger ones may be underutilized. LoRA converges faster, but Engram offers a safer learning path with better catastrophic forgetting resistance. As an open research project, ongoing development may introduce experimental changes.