Authored Projects (11)

rendergit by karpathy

2.8%

Flatten GitHub repos into a single HTML page

Browse code with syntax highlighting and sidebar navigation for easy exploration.
Copy entire codebases as CXML text for LLM analysis and code review.
Renders markdown and filters out binaries/large files for cleaner output.
Provides a responsive, search-friendly interface for quick code access.

Created 4 weeks ago

Updated 4 weeks ago

Starred by

Peter Norvig(Author of "Artificial Intelligence: A Modern Approach"; Research Director at Google),

Alexey Milovidov(Cofounder of Clickhouse),

Didier Lopes(Founder of OpenBB),

Yaowei Zheng(Author of LLaMA-Factory), and

27 more.

llm.c by karpathy

0.2%

28k

LLM training in pure C/CUDA, no PyTorch needed

Replicates GPT-2/3 pretraining; parallel PyTorch reference.
Faster than PyTorch; simple CPU fp32 reference (~1,000 lines).
Includes CUDA kernels, Flash Attention via cuDNN.
Supports multi-GPU/node training via MPI/NCCL.

Created 1 year ago

Updated 2 months ago

Starred by

Tobi Lutke(Cofounder of Shopify),

Daniel Gross(Cofounder of Safe Superintelligence),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Andrew Kane(Author of pgvector), and

44 more.

nanoGPT by karpathy

0.5%

44k

Minimalist repo for training/finetuning GPT models

Trains GPT-2 (124M) on OpenWebText using PyTorch DDP.
Includes scripts for training, finetuning, and sampling.
Supports single GPU, multi-GPU, and CPU training.
Uses PyTorch 2.0 torch.compile for potential speedup.

Created 2 years ago

Updated 9 months ago

Starred by

Zack Li(Cofounder of Nexa AI),

Didier Lopes(Founder of OpenBB),

Elvis Saravia(Founder of DAIR.AI),

Gabriel Almeida(Cofounder of Langflow), and

6 more.

nn-zero-to-hero by karpathy

0.9%

18k

Educational resource for neural network development, from basics to advanced models

Introduces backpropagation and training via micrograd, using Python and calculus.
Progresses to language modeling with torch.Tensor, training, and loss evaluation.
Explores MLPs, activations, gradients, BatchNorm, and deep network diagnostics.
Builds a GPT from scratch, covering tokenizers and the "Attention is All You Need" paper.

Created 3 years ago

Updated 1 year ago

Starred by

Aravind Srinivas(Cofounder of Perplexity),

Nat Friedman(Former CEO of GitHub),

George Hotz(Author of tinygrad; Founder of the tiny corp, comma.ai),

Lilian Weng(Cofounder of Thinking Machines Lab), and

31 more.

minGPT by karpathy

0.1%

23k

Minimal PyTorch re-implementation for GPT training and inference

Transformer model definition in ~300 lines of code.
Byte Pair Encoder translates text to integer sequences.
Includes GPT-independent PyTorch boilerplate training code.
Demos for addition, character-level language modeling, and text generation.

Created 5 years ago

Updated 1 year ago

Starred by

Carol Willing(Core Contributor to CPython, Jupyter) and

Eugene Yan(AI Scientist at AWS).

build-nanogpt by karpathy

0.2%

Educational resource for building nanoGPT from scratch

Walks through GPT-2 (124M) reproduction via step-by-step Git commits.
Includes a video lecture explaining each commit and code component.
Focuses on language model pretraining, not chat finetuning.

Created 1 year ago

Updated 1 year ago

Starred by

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA) and

Simon Willison(Coauthor of Django).

nano-llama31 by karpathy

Minimal Llama 3.1 implementation for training, finetuning, and inference

Reproduces reference output, ensuring a bug-free PyTorch adaptation.
Includes early draft of finetuning on Tiny Stories dataset.
Focuses on the 8B base model of Llama 3.1.

Created 1 year ago

Updated 1 year ago

Starred by

Vincent Weisser(Cofounder of Prime Intellect),

Robin Huang(Cofounder of Comfy Org),

Jared Palmer(Ex-VP AI at Vercel; Founder of Turborepo; Author of Formik, TSDX),

Russ d'Sa(Cofounder of LiveKit), and

16 more.

LLM101n by karpathy

0.1%

34k

Educational resource for building a Storyteller AI LLM

End-to-end construction of a ChatGPT-like web app from scratch.
Covers basics to advanced topics in Python, C, and CUDA.
Includes tokenization, optimization, and distributed training.
Explores finetuning (SFT, RLHF) and multimodal (VQVAE, diffusion).

Created 1 year ago

Updated 1 year ago

Starred by

Jeff Hammerbacher(Cofounder of Cloudera),

Carol Willing(Core Contributor to CPython, Jupyter),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA), and

11 more.

minbpe by karpathy

0.1%

10k

Minimal BPE encoder/decoder for LLM tokenization

Trains tokenizer vocab and merges on given text corpus.
Encodes text to tokens and decodes tokens back to text.
Implements basic, regex-based, and GPT-4-compatible tokenizers.
Includes special token handling, mirroring tiktoken's GPT-4 behavior.

Created 1 year ago

Updated 1 year ago

Starred by

Omar Sanseviero(DevRel at Google DeepMind)

Deshraj Yadav(Cofounder of Mem0)

Nathan Lambert(Research Scientist at AI2)

makemore by karpathy

0.2%

Character-level language model for generating text

Autoregressive model creates new "things" from a text file of training examples.
Supports bigram, MLP, CNN, RNN, LSTM, GRU, and Transformer architectures.
Generates samples or trains models; PyTorch is the only dependency.

Created 3 years ago

Updated 1 year ago

Starred by