Summary

This project implements Andrej Karpathy's microGPT, a character-level Transformer, entirely in Verilog RTL and deploys it on a Xilinx Virtex-5 FPGA. It targets engineers and researchers exploring hardware acceleration for AI models, offering a high-throughput, low-resource implementation of a neural network inference engine on custom silicon. The primary benefit is achieving approximately 50,000–69,000 tokens/second inference speed on an FPGA.

How It Works

The architecture employs a microcode-ROM sequencer driving modular datapath actuators, operating in Q5.11 fixed-point precision. A key optimization is incremental decoding with a persistent KV cache, which significantly reduces computation by processing only new tokens and attending over cached context. This design choice is crucial for achieving high inference throughput on hardware.

Quick Start & Requirements

• Hardware: Xilinx Virtex-5 FPGA (XC5VLX110T, XUPV5 / ML509 board).
• Software: Xilinx ISE 14.7 for synthesis/bitstream generation; Python 3 with numpy/torch for model export; Xilinx iSim for simulation.
• Build & Run: Model export via Python scripts (tools/train.py, tools/export.py, tools/ucode_asm.py). FPGA bitstream generation requires ISE tools with board/xupv5_microgpt_top.prj and board/xupv5_microgpt.ucf.
• Prerequisites: Specific FPGA board and legacy Xilinx ISE 14.7 toolchain are mandatory.

Highlighted Details

• Achieves ~50,000–69,000 tokens/second inference throughput at 80 MHz on the target FPGA.
• The core utilizes a microcode-ROM sequencer, modular datapath actuators, and Q5.11 fixed-point arithmetic.
• Resource utilization on XC5VLX110T is ~23% LUTs, 8% registers, and 96% DSP48E slices, with DSP being the primary bottleneck.
• Key engineering lessons highlight the significant performance gain from KV caching (3.2x) and the critical need to work around specific Xilinx XST synthesis tool bugs for reliable hardware bring-up.

Maintenance & Community

No specific community links, active maintenance indicators, or notable contributors are mentioned.

Licensing & Compatibility

The README does not specify a software license, potentially posing compatibility concerns for commercial use.

Limitations & Caveats

Requires specific, older hardware (Virtex-5 FPGA) and a legacy toolchain (Xilinx ISE 14.7). Several Xilinx XST synthesis tool bugs were encountered and require specific workarounds. The implementation is limited to a single transformer block and uses fixed-point arithmetic, which may impact accuracy compared to floating-point, though bit-exact replication to a Python reference is claimed.