Karpathy's LLM Lecture Becomes Interactive HTML Walkthrough
Get the Tech newsletter
Daily tech — startups, AI labs, chips, the launches that shape the next decade. Free.
- Andrej Karpathy's technical lecture on LLM training has been turned into an interactive HTML guide walking through the full pipeline from raw internet text to a conversational assistant
- Common Crawl has indexed 2.7 billion web pages since 2007, which gets filtered through URL blocklists, text extraction, language filtering (≥65% English), deduplication, and PII removal to produce the 44 TB FineWeb dataset of ~15 trillion tokens
- GPT-4 uses a vocabulary of 100,277 tokens built via Byte Pair Encoding, converting text into sub-word chunks that share roots like 'run' + 'ning' and 'run' + 'ner'
- Scaling laws show model accuracy is a smooth, predictable function of parameters (N) and training tokens (D) with no signs of plateauing — Llama 3 reached 405B parameters on 15T tokens, fueling a GPU arms race
- The base model after pre-training is a 'token simulator,' not an assistant — a lossy compression of the internet stored in its parameters, capable of verbatim Wikipedia recall but also plausible confabulation past its knowledge cutoff
- Supervised fine-tuning (SFT) using datasets like UltraChat — millions of conversations, mostly LLM-generated and human-reviewed — turns the base model into a chatbot by training it to imitate the 'ideal assistant' persona crafted by human labelers
Why it matters: Distilling a frontier LLM into an interactive walkthrough exposes the unglamorous mechanics behind ChatGPT: 15 trillion filtered tokens, hundreds of billions of parameters, and SFT datasets that train the model to imitate human labelers. The page makes visible why only a handful of well-funded labs can compete in the GPU arms race Karpathy describes.
Ask SkimNews
