<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> This repository provides the reference implementation for the Jacobian lens, a tool designed to interpret internal activations of large language models. It addresses the challenge of understanding what specific internal states "want" to say, targeting LLM interpretability researchers and engineers. The benefit lies in decoding these internal states into ranked vocabulary tokens, offering a direct window into the model's decision-making process.

How It Works

The Jacobian lens linearly transports residual-stream vectors from any layer to the final-layer basis using the average input-output Jacobian matrix (J_l = E[∂h_final / ∂h_l]) over a text corpus. The transported vector is then decoded via the model's unembedding layer, yielding ranked vocabulary tokens representing the activation's disposition. This offers a novel method to directly query and visualize the semantic content of internal model states.

Quick Start & Requirements

Installation: pip install -e .. Requires HuggingFace transformers models and typically a CUDA-enabled GPU. Fitting a lens involves providing model, prompts, and checkpoint paths; quality reportedly saturates quickly (~100 prompts). The walkthrough.ipynb notebook offers an end-to-end guide.

Highlighted Details

• Features an interactive layer × position view visualizing lens outputs, showing top-1 words at specific activation points.
• Demonstrates interpretability with examples like the ASCII-face, correctly identifying "nose" at mid-layers even when absent from the prompt.
• Designed for compatibility with various HuggingFace decoder-only transformers.
• Fitting requires minimal data; quality stabilizes rapidly with ~100 prompts.

Maintenance & Community

This project is explicitly marked as "Not maintained and not accepting contributions." No community channels are listed.

Licensing & Compatibility

Code and prompt sets are released under the permissive Apache License 2.0, generally allowing commercial use. The library adapts cleanly to other HuggingFace decoders. Note that model weights and datasets downloaded at runtime are subject to their respective licenses.

Limitations & Caveats

The project is "Not maintained and not accepting contributions," indicating a lack of ongoing development or support. It is a reference implementation and is not optimized for performance, with fitting times potentially dominated by the model's backward pass.