Authored Projects (18)

Starred by

George Hotz(Author of tinygrad; Founder of the tiny corp, comma.ai),

Rotem Weiss(Cofounder of Tavily),

Elie Bursztein(Cybersecurity Lead at Google DeepMind),

Eric Zhang(Founding Engineer at Modal), and

26 more.

nanochat by karpathy

1.2%

52k

A minimal, full-stack LLM implementation for accessible AI development

Trains, fine-tunes, and serves ChatGPT-like models end-to-end on single GPU nodes.
Features a dependency-lite, hackable PyTorch codebase for research and learning.
Enables low-cost ($100 tier) LLM training and inference with gradient accumulation.
Packages code for querying by other LLMs, demonstrating AI-to-AI interaction.

Created 6 months ago

Updated 2 weeks ago

Starred by

Shawn Wang(Editor of Latent Space),

Rodrigo Nader(Cofounder of Langflow),

Didier Lopes(Founder of OpenBB),

Bryan Helmig(Cofounder of Zapier), and

22 more.

autoresearch by karpathy

6.8%

70k

Autonomous LLM research agent for single-GPU training

Automates LLM pretraining experiments by iteratively modifying and training models.
Leverages PyTorch for GPT model training with a fixed time budget and BPB metric.
Agent-driven code modification and hyperparameter tuning for efficient research exploration.
Enables overnight AI-driven research, waking to improved models and experiment logs.

Created 1 month ago

Updated 2 weeks ago

jobs by karpathy

8.0%

Visualizing US job market data with LLM-driven insights

Interactive treemap visualization of US job market data, exploring employment, pay, and education.
LLM-powered coloring to assess AI exposure and other custom criteria across occupations.
Data pipeline integrates scraping, parsing, and LLM scoring for dynamic analysis.
Forkable prompt system allows custom LLM analysis for diverse impact assessments.

Created 4 weeks ago

Updated 3 weeks ago

rustbpe by karpathy

2.9%

424

Efficient Rust library for BPE tokenizer training

Trains GPT-style Byte Pair Encoding tokenizers, filling a gap in existing libraries.
Exports directly to tiktoken format for fast, optimized inference.
Features parallel processing via Rust and rayon for high-speed training.
Offers Python bindings through PyO3 for easy integration into Python workflows.

Created 3 months ago

Updated 3 months ago

Starred by

Vaibhav Nivargi(Cofounder of Moveworks).

hn-time-capsule by karpathy

1.5%

598

Analyzing historical discussions with LLMs for prescience

Fetches decade-old Hacker News articles and comments for retrospective analysis.
Leverages LLMs to evaluate commenter prescience and predict future outcomes.
Generates HTML reports summarizing insights and grading commenter accuracy.
Synthesizes knowledge from past human artifacts using AI.

Created 4 months ago

Updated 4 months ago

Starred by

Shane Thomas(Cofounder of Mastra),

Dan Guido(Cofounder of Trail of Bits),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Jack Lukic(Author of Semantic UI), and

3 more.

llm-council by karpathy

2.0%

17k

A multi-LLM collaborative framework for enhanced question answering

Query multiple LLMs simultaneously, then have them review and rank each other's outputs.
A designated "Chairman" LLM synthesizes peer-reviewed responses into a final, consolidated answer.
Built with FastAPI and React, leveraging OpenRouter for API aggregation and model management.
Facilitates side-by-side LLM evaluation and exploration for complex query resolution.

Created 4 months ago

Updated 4 months ago

Starred by

Shyamal Anadkat(Research Scientist at OpenAI) and

Travis Fischer(Founder of Agentic).

reader3 by karpathy

1.7%

Self-hosted EPUB reader for LLM interaction

Enables chapter-by-chapter reading of EPUBs for easy copy-pasting into LLMs.
Lightweight application for accessing and interacting with digital book content.
Simple Python tool using uv for straightforward execution and server setup.
Facilitates collaborative reading by integrating book content with large language models.

Created 4 months ago

Updated 4 months ago

Starred by

Tobi Lutke(Cofounder of Shopify),

Daniel Gross(Cofounder of Safe Superintelligence),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Andrew Kane(Author of pgvector), and

44 more.

nanoGPT by karpathy

0.8%

56k

Minimalist repo for training/finetuning GPT models

Trains GPT-2 (124M) on OpenWebText using PyTorch DDP.
Includes scripts for training, finetuning, and sampling.
Supports single GPU, multi-GPU, and CPU training.
Uses PyTorch 2.0 torch.compile for potential speedup.

Created 3 years ago

Updated 5 months ago

rendergit by karpathy

1.8%

Flatten GitHub repos into a single HTML page

Browse code with syntax highlighting and sidebar navigation for easy exploration.
Copy entire codebases as CXML text for LLM analysis and code review.
Renders markdown and filters out binaries/large files for cleaner output.
Provides a responsive, search-friendly interface for quick code access.

Created 7 months ago

Updated 7 months ago

Starred by

Peter Norvig(Author of "Artificial Intelligence: A Modern Approach"; Research Director at Google),

Alexey Milovidov(Cofounder of Clickhouse),

Didier Lopes(Founder of OpenBB),

Yaowei Zheng(Author of LLaMA-Factory), and

27 more.

llm.c by karpathy

0.5%

29k

LLM training in pure C/CUDA, no PyTorch needed

Replicates GPT-2/3 pretraining; parallel PyTorch reference.
Faster than PyTorch; simple CPU fp32 reference (~1,000 lines).
Includes CUDA kernels, Flash Attention via cuDNN.
Supports multi-GPU/node training via MPI/NCCL.

Created 2 years ago

Updated 9 months ago

Starred by

Zack Li(Cofounder of Nexa AI),

Didier Lopes(Founder of OpenBB),

Elvis Saravia(Founder of DAIR.AI),

Gabriel Almeida(Cofounder of Langflow), and

6 more.

nn-zero-to-hero by karpathy

0.7%

21k

Educational resource for neural network development, from basics to advanced models

Introduces backpropagation and training via micrograd, using Python and calculus.
Progresses to language modeling with torch.Tensor, training, and loss evaluation.
Explores MLPs, activations, gradients, BatchNorm, and deep network diagnostics.
Builds a GPT from scratch, covering tokenizers and the "Attention is All You Need" paper.

Created 3 years ago

Updated 1 year ago

Starred by

Aravind Srinivas(Cofounder of Perplexity),

Nat Friedman(Former CEO of GitHub),

George Hotz(Author of tinygrad; Founder of the tiny corp, comma.ai),

Lilian Weng(Cofounder of Thinking Machines Lab), and

31 more.

minGPT by karpathy

0.3%

24k

Minimal PyTorch re-implementation for GPT training and inference

Transformer model definition in ~300 lines of code.
Byte Pair Encoder translates text to integer sequences.
Includes GPT-independent PyTorch boilerplate training code.
Demos for addition, character-level language modeling, and text generation.

Created 5 years ago

Updated 1 year ago

Starred by

Carol Willing(Core Contributor to CPython, Jupyter) and

Eugene Yan(AI Scientist at AWS).

build-nanogpt by karpathy

0.5%

Educational resource for building nanoGPT from scratch

Walks through GPT-2 (124M) reproduction via step-by-step Git commits.
Includes a video lecture explaining each commit and code component.
Focuses on language model pretraining, not chat finetuning.

Created 1 year ago

Updated 1 year ago

Starred by

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA) and

Simon Willison(Coauthor of Django).

nano-llama31 by karpathy

Minimal Llama 3.1 implementation for training, finetuning, and inference

Reproduces reference output, ensuring a bug-free PyTorch adaptation.
Includes early draft of finetuning on Tiny Stories dataset.
Focuses on the 8B base model of Llama 3.1.

Created 1 year ago

Updated 1 year ago

Starred by

Vincent Weisser(Cofounder of Prime Intellect),

Robin Huang(Cofounder of Comfy Org),

Jared Palmer(SVP at GitHub; Founder of Turborepo; Author of Formik, TSDX),

Russ d'Sa(Cofounder of LiveKit), and

16 more.

LLM101n by karpathy

0.3%

37k

Educational resource for building a Storyteller AI LLM

End-to-end construction of a ChatGPT-like web app from scratch.
Covers basics to advanced topics in Python, C, and CUDA.
Includes tokenization, optimization, and distributed training.
Explores finetuning (SFT, RLHF) and multimodal (VQVAE, diffusion).

Created 1 year ago

Updated 1 year ago

Starred by

Jeff Hammerbacher(Cofounder of Cloudera),

Carol Willing(Core Contributor to CPython, Jupyter),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA), and

12 more.

minbpe by karpathy

0.1%

10k

Minimal BPE encoder/decoder for LLM tokenization

Trains tokenizer vocab and merges on given text corpus.
Encodes text to tokens and decodes tokens back to text.
Implements basic, regex-based, and GPT-4-compatible tokenizers.
Includes special token handling, mirroring tiktoken's GPT-4 behavior.

Created 2 years ago

Updated 1 year ago

Starred by

Omar Sanseviero(DevRel at Google DeepMind)

Deshraj Yadav(Cofounder of Mem0)

Nathan Lambert(Research Scientist at AI2)

makemore by karpathy

0.7%

Character-level language model for generating text

Autoregressive model creates new "things" from a text file of training examples.
Supports bigram, MLP, CNN, RNN, LSTM, GRU, and Transformer architectures.
Generates samples or trains models; PyTorch is the only dependency.

Created 3 years ago

Updated 1 year ago

Starred by