Phil Wang

Prolific Research Paper Implementer

Authored Projects (29)

Starred by

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Jeff Hammerbacher(Cofounder of Cloudera),

Chenlin Meng(Cofounder of Pika),

Jiayi Pan(Author of SWE-Gym; MTS at xAI), and

8 more.

x-transformers by lucidrains

0.2%

Transformer library with extensive experimental features

Implements various attention mechanisms like FlashAttention and Rotary Positional Embeddings.
Supports advanced normalization techniques (RMSNorm, ScaleNorm, SandwichNorm) for training stability.
Integrates memory mechanisms and sparse attention for efficient long-context processing.
Offers multimodal capabilities, including image-to-text generation with Vision Transformers.

Created 4 years ago

Updated 5 days ago

Starred by

Pawel Garbacki(Cofounder of Fireworks AI),

Wing Lian(Founder of Axolotl AI),

Omar Sanseviero(DevRel at Google DeepMind),

Jeffrey Quesnelle(Cofounder of Nous Research), and

6 more.

PaLM-rlhf-pytorch by lucidrains

0.1%

RLHF implementation on PaLM

Trains PaLM with Reinforcement Learning from Human Feedback.
Includes reward modeling with optional LoRA finetuning.
Uses PPO for RLHF training loop.
Supports FlashAttention for faster training.

Created 2 years ago

Updated 2 weeks ago

Starred by

Johannes Hagemann(Cofounder of Prime Intellect).

clinical-calculator-tooluse by lucidrains

315

ChatGPT fine-tune for primary care medicine

Collects patient histories and suggests differential diagnoses.
Aims for specialist knowledge via fine-tuning.
Scribing for basic medical cases.

Created 2 years ago

Updated 2 weeks ago

Starred by

Jiayi Pan(Author of SWE-Gym; MTS at xAI)

Yang Song(Professor at Caltech; Research Scientist at OpenAI)

Benjamin Bolte(Cofounder of K-Scale Labs)

vector-quantize-pytorch by lucidrains

0.5%

PyTorch library for vector quantization techniques

Vector quantization with exponential moving averages for dictionary updates.
Includes residual VQ, grouped residual VQ, and cosine similarity options.
Offers codebook initialization via k-means and gradient rotation trick.
Implements lookup-free quantization (LFQ) and latent quantization methods.

Created 5 years ago

Updated 2 weeks ago

Starred by

Andrej Karpathy(Founder of Eureka Labs; Formerly at Tesla, OpenAI; Author of CS 231n),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Forrest Iandola(Author of SqueezeNet; Research Scientist at Meta),

Yaowei Zheng(Author of LLaMA-Factory), and

10 more.

vit-pytorch by lucidrains

0.2%

24k

PyTorch library for Vision Transformer variants and related techniques

ViT, SimpleViT, and many variants incl. DeepViT, CaiT, CrossViT, PiT, LeViT, CvT, Twins, RegionViT.
Includes masked autoencoders, masked image modeling, and masked patch prediction.
Supports 3D ViT for video, medical imaging, and ViViT for efficient video attention.
Offers Dino and EsViT self-supervised learning techniques, plus attention/embedding access.

Created 5 years ago

Updated 1 month ago

Starred by

Wing Lian(Founder of Axolotl AI).

native-sparse-attention-pytorch by lucidrains

0.7%

744

Sparse attention implementation from Deepseek's research paper

Hardware-aligned sparse attention, natively trainable.
Uses Triton and CUDA kernel hacking.
Includes example Enwik8 language modeling.

Created 7 months ago

Updated 1 month ago

rotary-embedding-torch by lucidrains

0.4%

750

PyTorch library for rotary embeddings in transformers

Implements rotary positional encoding for enhanced transformer models.
Includes axial rotary embeddings for n-dimensional data like video.
Offers length extrapolation via XPos for autoregressive transformers.
Supports sequence position interpolation for longer context windows.

Created 4 years ago

Updated 1 month ago

Starred by

Ross Wightman(Author of timm; CV at Hugging Face).

transfusion-pytorch by lucidrains

0.2%

Pytorch implementation for multimodal model research

Predicts next token, diffuses images; substitutes diffusion with flow matching.
Handles multiple modalities: text, images, audio, etc.
Uses transformer architecture; encodes/decodes images automatically.

Created 1 year ago

Updated 3 months ago

titans-pytorch by lucidrains

0.5%

PyTorch module for memory-efficient Transformers, based on the Titans paper

Implements neural memory module with configurable chunk size for performance tuning.
Includes Memory-As-Context Transformer with local & long-term memory tokens.
Supports training and sampling from the transformer architecture.

Created 8 months ago

Updated 3 months ago

nGPT-pytorch by lucidrains

290

PyTorch implementation of nGPT, a normalized GPT learning on the hypersphere

Explores representation learning via hypersphere normalization in Transformers.
Includes flag for QK normalization in attention mechanisms.
Shows potential for continual learning and plasticity.
Applied to improve UTD for off-policy reinforcement learning.

Created 11 months ago

Updated 3 months ago

Starred by

Alex Yu(Research Scientist at OpenAI; Former Cofounder of Luma AI)

Yineng Zhang(Inference Lead at SGLang; Research Scientist at Together AI)

Vincent Weisser(Cofounder of Prime Intellect)

ring-attention-pytorch by lucidrains

0.2%

538

Pytorch impl of Ring Attention for near-infinite context

Splits data across sequence dimension, uses ring reduce for attention matrix tiles.
Includes Striped Attention, permuting sequences for better workload balancing.
Supports grouped query attention, saving communication costs during ring reduce.
Features Tree Attention Decoding for topology-aware long-context attention.

Created 1 year ago

Updated 4 months ago

Starred by

Shawn Wang(Editor of Latent Space).

soundstorm-pytorch by lucidrains

0.1%

Pytorch implementation of SoundStorm for efficient parallel audio generation

Uses MaskGiT on residual vector quantized codes from Soundstream.
Employs a Conformer architecture tailored for the audio domain.
Supports training on raw audio via integration with a pre-trained SoundStream.
Includes text-to-speech functionality with TextToSemantic encoder/decoder.

Created 2 years ago

Updated 4 months ago

Starred by

Benjamin Bolte(Cofounder of K-Scale Labs) and

Omar Sanseviero(DevRel at Google DeepMind).

audiolm-pytorch by lucidrains

0.3%

PyTorch implementation of Google's AudioLM for audio generation

Uses hierarchical transformers (semantic, coarse, fine) for audio modeling.
Employs SoundStream/EnCodec for audio tokenization and reconstruction.
Supports text-to-audio synthesis via classifier-free guidance with T5.
Includes training and inference code for all components, plus multi-GPU support.

Created 3 years ago

Updated 8 months ago

Starred by

Luis Capelo(Cofounder of Lightning AI),

Vincent Weisser(Cofounder of Prime Intellect),

Patrick von Platen(Author of Hugging Face Diffusers; Research Engineer at Mistral), and

Alexander Borzunov(Research Scientist at OpenAI).

speculative-decoding by lucidrains

285

Speculative decoding explorations

Implements early exit scheme, caching hidden layers for faster speculative decoding.
Batched speculative decoding allows variable advancement rates across rows.
Includes speculative sampling with prophet transformer for potential speedup.
Aims to accelerate text-to-semantic decoder in Spear-TTS.

Created 2 years ago

Updated 9 months ago

Starred by

Benjamin Bolte(Cofounder of K-Scale Labs) and

Ross Wightman(Author of timm; CV at Hugging Face).

robotic-transformer-pytorch by lucidrains

0.5%

441

PyTorch implementation of RT1 (Robotic Transformer) for robotic control

Uses video frames and instructions to predict action logits.
Employs MaxViT as the vision transformer backbone.
Includes classifier-free guidance with conditional scaling.

Created 2 years ago

Updated 11 months ago

voicebox-pytorch by lucidrains

661

Pytorch implementation of MetaAI's Voicebox text-to-speech model

Uses conditional flow matching for training a TTS system.
Integrates with Spear-TTS for text-to-semantic conversion.
Employs adaptive normalization and can condition on text.
Includes Encodec/Voco for audio encoding/decoding.

Created 2 years ago

Updated 11 months ago

Starred by

Andreas Jansson(Cofounder of Replicate),

Wing Lian(Founder of Axolotl AI),

Max Howell(Author of Homebrew),

Eric Zhu(Coauthor of AutoGen; Research Scientist at Microsoft Research), and

3 more.

toolformer-pytorch by lucidrains

0.1%

Pytorch implementation of Toolformer for language models using external tools

Fine-tunes a language model to make API calls based on perplexity reduction.
Includes filter_tokens_with_api_response for filtering model outputs.
Supports custom tools via a function registry for API call execution.

Created 2 years ago

Updated 1 year ago

Starred by

Benjamin Bolte(Cofounder of K-Scale Labs),

Victor Taelin(Author of Bend, Kind, HVM),

Jesse Clark(Cofounder of Marqo),

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA), and

10 more.

DALLE2-pytorch by lucidrains

0.1%

11k

PyTorch implementation of DALL-E 2, a text-to-image synthesis neural network

Diffusion prior network predicts image embedding based on CLIP text embedding.
Includes Unet, Decoder, CLIP, and DiffusionPrior for image generation.
Supports cascading DDPMs, latent diffusion, and OpenAI CLIP integration.
Offers inpainting using a resampler from the Repaint paper.

Created 3 years ago

Updated 1 year ago

Starred by

Chenlin Meng(Cofounder of Pika).

video-diffusion-pytorch by lucidrains

PyTorch implementation of video diffusion models

Generates videos from text using a space-time factored U-Net.
Supports text conditioning via BERT embeddings.
Includes a Trainer class for training on folders of GIFs.
Enables co-training of images and video with attention control.

Created 3 years ago

Updated 1 year ago

Starred by

Vincent Weisser(Cofounder of Prime Intellect),

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Pawel Garbacki(Cofounder of Fireworks AI),

Shawn Wang(Editor of Latent Space), and

4 more.

self-rewarding-lm-pytorch by lucidrains

0.1%

Training framework for self-rewarding language models

Implements Self-Rewarding Training and SPIN (Self-Play Imitation) fine-tuning.
Supports custom reward prompts beyond LLM-as-judge via regex parsing.
Enables flexible fine-tuning orders: SFT, SPIN, external/self-rewarding DPO.
Uses X-Transformers; checkpoints saved after each fine-tuning stage.

Created 1 year ago

Updated 1 year ago

Starred by

Elvis Saravia(Founder of DAIR.AI),

Luis Capelo(Cofounder of Lightning AI),

Ben Firshman(Cofounder of Replicate),

Chenlin Meng(Cofounder of Pika), and

8 more.

DALLE-pytorch by lucidrains

0.0%

PyTorch implementation of DALL-E, OpenAI's text-to-image transformer

Trains a discrete VAE to compress images into discrete visual tokens.
Uses a multimodal transformer to generate images from text.
Supports OpenAI's pretrained VAE and Taming Transformer's VQGAN.
Includes CLIP model for ranking generated images.

Created 4 years ago

Updated 1 year ago

Starred by

Benjamin Bolte(Cofounder of K-Scale Labs).

spear-tts-pytorch by lucidrains

273

Pytorch implementation of a multi-speaker text-to-speech attention network

Text-to-semantic module for SoundStorm conditioning.
Uses grouped query attention for memory-efficient decoding.
Includes speculative decoding with early exit strategy.
Features flexible freezing of encoder/decoder layers during training.

Created 2 years ago

Updated 1 year ago

Starred by

Jesse Clark(Cofounder of Marqo),

Jeffrey Quesnelle(Cofounder of Nous Research),

Philipp Schmid(DevRel at Google DeepMind), and

Andreas Blattmann(Cofounder of Black Forest Labs).

RETRO-pytorch by lucidrains

871

PyTorch implementation of Deepmind's RETRO retrieval-augmented LM

Uses Faiss/autofaiss for fast k-NN retrieval over indexed text chunks.
Leverages rotary embeddings for relative positional encoding.
Includes training wrapper for processing text documents into memmapped arrays.
Supports DeepNet scaling techniques for training 1000+ layer models.

Created 3 years ago

Updated 1 year ago

Starred by

Anastasios Angelopoulos(Cofounder of LMArena).

naturalspeech2-pytorch by lucidrains

0.1%

PyTorch implementation of Natural Speech 2, a zero-shot speech/singing synthesizer

Uses neural audio codec with continuous latent vectors for natural speech synthesis.
Leverages latent diffusion model with non-autoregressive generation.
Includes improvements for attention/transformer components.
Offers speech prompt encoders and multilingual phonemizer/aligner.

Created 2 years ago

Updated 2 years ago

Starred by

Chenlin Meng(Cofounder of Pika),

Haotian Liu(Author of LLaVA; Research Scientist at xAI),

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA),

Thomas Wolf(Cofounder of Hugging Face), and

3 more.

reformer-pytorch by lucidrains

0.1%

PyTorch implementation of the Reformer, an efficient Transformer research paper

Includes LSH attention, reversible networks, and chunking for memory efficiency.
Supports masking, axial position embeddings, and integration with DeepSpeed.
Offers helpers for auto-regressive language models and sequence padding.
Provides a wrapper for simplified encoder/decoder architecture creation.

Created 5 years ago

Updated 2 years ago

Starred by

Vincent Weisser(Cofounder of Prime Intellect),

Stella Rose Biderman(Executive Director at EleutherAI),

Jeff Hammerbacher(Cofounder of Cloudera),

Anastasios Angelopoulos(Cofounder of LMArena), and

1 more.

alphafold2 by lucidrains

Pytorch implementation for protein structure prediction

Predicts distograms, angles, and atomic coordinates from sequence and MSA data.
Uses SE3 Transformer, E(n)-Transformer, or EGNN for structural refinement.
Supports MSA, ESM, and ProtTrans embeddings; convolutions; sparse/linear/Kronecker attention.
Includes template processing with axial attention and sidechain information.

Created 4 years ago

Updated 2 years ago

Starred by

Paras Jain(Cofounder of Genmo).

triton-transformer by lucidrains

274

Transformer implementation in Triton

Trains vanilla transformer models using Triton for efficiency.
Supports causal (autoregressive) and non-causal attention.
Includes layernorm, softmax, and cross-entropy implemented with Triton ops.

Created 4 years ago

Updated 3 years ago

Starred by

Luis Capelo(Cofounder of Lightning AI),

Paras Jain(Cofounder of Genmo),

Johannes Hagemann(Cofounder of Prime Intellect),

Jane Manchun Wong(Security Researcher), and

1 more.

deep-daze by lucidrains

CLI tool for text-to-image generation using CLIP and SIREN

Generates images from text prompts using OpenAI's CLIP model.
Leverages SIREN (implicit neural representation network) for image creation.
Optimizes images based on text or image inputs, or a combination.
Supports story creation by optimizing on sliding windows of input text.

Created 4 years ago

Updated 3 years ago

Starred by

Max Howell(Author of Homebrew)

Shizhe Diao(Author of LMFlow; Research Scientist at NVIDIA)

Andreas Jansson(Cofounder of Replicate)

big-sleep by lucidrains

CLI tool for text-to-image generation

Uses OpenAI's CLIP and BigGAN to generate images from natural language prompts.
Allows training on multiple phrases with optional penalization of unwanted features.
Offers options for saving progress, best images, and using a larger vision model.
Enables experimentation with the number of classes used for BigGAN.

Created 4 years ago

Updated 3 years ago