Most deep learning progress used to read like a sequence of lucky architectural bets. The quieter, more consequential claim here is that for language models the gains are so regular you can write them down as equations. Cross-entropy loss falls as a power law in three knobs — parameters, data, and compute — with the same trend holding across more than seven orders of magnitude. Once performance is that predictable, the central question stops being "what clever model should we build?" and becomes "given a fixed compute budget, how should we spend it?"
Key Findings
- Architecture matters less than scale. Within a wide range, width-versus-depth and other shape choices barely move the loss; the three scale variables dominate. That is why the field's energy shifted from architecture search toward simply scaling a known-good design.
- Bigger models are more sample-efficient. Large models reach a given loss after seeing fewer tokens, so the compute-optimal recipe is to train a very large model on a relatively modest amount of data — and to stop well before convergence rather than squeezing every epoch.
- You can predict before you train. The fitted laws let you extrapolate the loss of a model far larger than anything you have run, which is precisely what made committing a vast compute budget to GPT-3 a calculated bet instead of a gamble.
Great Fit / When to Skip
Great fit if you want the empirical backbone behind the entire scaling era, or you need to reason about compute-optimal allocation rather than chase benchmarks. Skip, or read alongside newer work, if you want the final word on optimal data-versus-parameter ratios: Chinchilla (2022) later argued these laws under-trained on data, shifting the compute-optimal frontier toward more tokens. Read this as the foundation that opened the question, not the closed answer.