Let's reproduce GPT-2 (124M) - YouTube

Tutorial  

We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar.

Links: - build-nanogpt GitHub repo, with all the changes in this video as individual commits: https://github.com/karpathy/build-nan... - nanoGPT repo: https://github.com/karpathy/nanoGPT - llm.c repo: https://github.com/karpathy/llm.c - my website: https://karpathy.ai - my twitter:   / karpathy   - our Discord channel:   / discord  

Supplementary links: - Attention is All You Need paper: https://arxiv.org/abs/1706.03762 - OpenAI GPT-3 paper: https://arxiv.org/abs/2005.14165 - OpenAI GPT-2 paper: https://d4mucfpksywv.cloudfront.net/b... The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to: https://lambdalabs.com

Chapters:

• 00:00:00 intro: Let’s reproduce GPT-2 (124M)
• 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint
• 00:13:47 SECTION 1: implementing the GPT-2 nn.Module
• 00:28:08 loading the huggingface/GPT-2 parameters
• 00:31:00 implementing the forward pass to get logits
• 00:33:31 sampling init, prefix tokens, tokenization
• 00:37:02 sampling loop
• 00:41:47 sample, auto-detect the device
• 00:45:50 let’s train: data batches (B,T) → logits (B,T,C)
• 00:52:53 cross entropy loss
• 00:56:42 optimization loop: overfit a single batch
• 01:02:00 data loader lite
• 01:06:14 parameter sharing wte and lm_head
• 01:13:47 model initialization: std 0.02, residual init
• 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms
• 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms
• 01:39:38 float16, gradient scalers, bfloat16, 300ms
• 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms
• 02:00:18 flash attention, 96ms
• 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms
• 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping
• 02:21:06 learning rate scheduler: warmup + cosine decay
• 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms
• 02:34:09 gradient accumulation
• 02:46:52 distributed data parallel (DDP)
• 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU)
• 03:23:10 validation data split, validation loss, sampling revive
• 03:28:23 evaluation: HellaSwag, starting the run
• 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro
• 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA
• 03:59:39 summary, phew, build-nanogpt github repo

Corrections: I will post all errata and followups to the build-nanogpt GitHub repo (link above)

SuperThanks: I experimentally enabled them on my channel yesterday. Totally optional and only use if rich. All revenue goes to to supporting my work in AI + Education.