How large language models work
Tokens, embeddings, attention, context windows, temperature and training: what is actually happening when a model answers you.
7 lessons · about 1 hour
What’s inside
- 01Tokens: how a model reads text
Models never see your words, only numbered chunks called tokens, and that explains odd spelling mistakes, surprise costs and hard limits. · 10 min
- 02Embeddings: meaning as position
Models turn each token into a long list of numbers that places it in a space where similar meanings sit close together. · 9 min
- 03The transformer and attention
The 2017 architecture behind nearly every modern language model, and the attention mechanism that lets every token weigh every other. · 11 min
- 04Pre-training: learning by predicting the next token
One simple objective, repeated across trillions of tokens, produces most of what a model knows and much of what it gets wrong. · 10 min
- 05Post-training: from text predictor to assistant
Instruction tuning and preference learning turn a raw text predictor into a helpful assistant, and explain its habits, good and bad. · 10 min
- 06Context windows, sampling and temperature
What a model can see at once, and how it chooses each word: two settings behind a surprising number of bugs. · 11 min
- 07Reasoning models, and why hallucination is built in
How models that think before answering use extra computation, and why making things up follows directly from how language models are built. · 12 min
