Measure the model's size correctly, and the scaling disagreement disappears.
Earlier scaling-law papers disagreed loudly on one practical question: given more compute, should you spend it on a bigger model or on more data? The disagreement was usually waved away as “different datasets.” DeepSeek pinned it to something concrete and fixable: everyone was measuring model size with raw parameter count, which quietly mis-counts the real cost of running the model.
Swap parameter count for non-embedding FLOPs per token — literally, the arithmetic the model does to read one token — and the conflicting results line up. The corrected law then says how to split a budget, lets you fix learning rate and batch size before a run, and predicts the loss of a giant run from cheap small ones. That is the “longtermism”: use small experiments to de-risk a single multi-million-GPU-hour bet.
A scaling law is only as good as the ruler it's drawn against. Fix the ruler, and a thousandfold extrapolation becomes a straight line you can trust.
The puzzle
You have a fixed pile of compute. Bigger model, or more data?
Step 1 · The black box & the one bet
A language model is a box you grade with one number — and you only get to train it once.
Here is the whole subject in one picture. A decoder-only Transformer reads text and predicts the next token (a sub-word chunk); do that across a corpus and you can score it. The score is the loss — how surprised the model is by the real next token. Lower is better. DeepSeek measures it as bits-per-byte on a held-out validation set, and that one number follows us through every step:
And here is the bind that makes this a research problem at all. A frontier run is a single, enormous, one-shot bet. You cannot tune it by trial and error — there is no “try again.” So the real question isn't “what's the best model?” It's: can you predict the best configuration cheaply, in advance, and be right?
One number, the loss in bits-per-byte, grades the whole model. Everything that follows is about predicting how low that number will go — before you commit the budget.
From the abstract: scaling laws in prior literature “present varying conclusions, which casts a dark cloud over scaling LLMs.” DeepSeek sets out to clear that cloud, then trains 7B and 67B base + chat models on a 2-trillion-token bilingual corpus.
Step 2 · Three quantities, one identity, one quarrel
Compute = model scale × data. The whole quarrel is what “model scale” means.
You've met the loss L and the one-shot bet. Now: what are you actually splitting? All of scaling rests on three quantities tied by a single identity.
Define each plainly — every law from here uses these three letters:
quantity 1 · compute
C
C — the compute budget. The total arithmetic you can afford. This is the thing held fixed; the question is how to spend it.
quantity 2 · model scale
M
M — model scale. How much model you build. Earlier work used parameter count N for this; DeepSeek's fix is to measure M differently — that's Step 3.
quantity 3 · data
D
D — data, in tokens. How much text the model trains on. DeepSeek's final corpus is 2 trillion tokens, mostly English and Chinese.
C=M·D
spend your compute C on a model of scale M, run over D tokens — that's the identity everything balances on
The old shorthand was C ≈ 6·N·D — six floating-point operations per
parameter per token. Convenient, but it bakes in raw parameter count N as the
measure of model scale. And that is exactly where the field's disagreement was
hiding. Look how far apart prior papers landed on the “bigger model vs. more data” split:
OpenAI (Kaplan 2020) said add compute mostly to the model: model exponent a ≈ 0.73, data exponent b ≈ 0.27. Chinchilla (Hoffmann 2022) said split it nearly evenly: a ≈ 0.49, b ≈ 0.51. Same goal, opposite advice — and billions of dollars of training ride on which is right.
Where Nopt ∝ Ca and Dopt ∝ Cb give the optimal model and data for a budget (a + b ≈ 1). The conflict between a ≈ 0.73 and a ≈ 0.49 is the “dark cloud” this paper sets out to clear (Table 4).
The fix
A better ruler for model size · set knobs in advance · ride a power law · and why data quality bends it
Step 3 · The better ruler — non-embedding FLOPs/token
Count the work, not the weights — and a 50% mismeasurement vanishes.
The quarrel hid inside the word “model scale.” So replace the loose ruler with a precise one — and do the arithmetic by hand on a small model to feel why it matters.
DeepSeek measures model scale as M, non-embedding FLOPs per token — the floating-point operations the Transformer body does to process one token, including attention but excluding the cheap vocabulary lookups. Compare three rulers for the same model. Take a small one with nlayer = 8, dmodel = 512, vocab 102,400, sequence length 4,096, and plug into the paper's formulas:
Ruler 6N₁ — non-embedding params only
72·nlayer·dmodel2. Counts the body's weights, ignores attention's own arithmetic.
Ruler 6N₂ — complete params (adds vocabulary)
Adds 6·nvocab·dmodel. The vocabulary barely adds model capacity, yet inflates the count a lot at small scale.
Ruler M — non-embedding FLOPs/token (the fix)
Adds attention's real cost 12·nlayer·dmodel·lseq, drops the vocabulary. This is what the model actually computes per token.
For this small model the “params” ruler 6N₁ reads 151M while the honest ruler M reads 352M — 6N₁ is only 0.43× of the truth, off by more than 50%. Use a ruler that's wrong by 50% and every point you fit the scaling curve to is shifted — which is precisely how two careful labs reached opposite conclusions.
Formulas (Eq. 2): 6N₁ = 72·nlayer·dmodel2; 6N₂ = 6N₁ + 6·nvocab·dmodel; M = 6N₁ + 12·nlayer·dmodel·lseq. The 6N₁/M ratio (Table 3) climbs from 0.43 at this scale toward 0.92 only at ~64B params — the error is worst exactly where the cheap experiments live.
Step 4 · Set the dials before you spend
Learning rate and batch size also follow power laws — so you can set them in advance.
With an honest ruler for M, you can fit clean curves. But a giant run also needs a learning rate and batch size — and you can't grid-search those at full scale. DeepSeek found you don't have to.
Across many small runs, the near-optimal batch size (how many tokens per update) and learning rate (how big each update step is) trace smooth power laws in the compute budget C. As C grows, the best batch size rises and the best learning rate falls — both predictably. So you read them off a fitted line instead of tuning by trial and error:
learning rate
ηopt = 0.3118 · C−0.1250
bigger budget → gentler steps
Read η straight off the line for any budget.
batch size
Bopt = 0.2920 · C0.3271
bigger budget → larger batches
Set B from the budget — no grid search.
a wide safe band
±0.25% loss
near-optimal is easy to hit
A broad band stays within 0.25% of best.
The hyperparameters you'd normally babysit become a lookup. And the band of near-optimal settings is wide, so you don't need to hit them exactly — you just need to not be wildly off. The expensive run starts already well-tuned.
Eq. 1, fitted on compute budgets from 1e17 to 2e19, then validated at 1e20. A setting was counted “near-optimal” if its generalization error exceeded the minimum by no more than 0.25%.
Step 5 · The allocation law — the centerpiece
For each extra unit of compute, grow the model and the data by fixed, near-equal shares.
Honest ruler, dials pre-set. Now the question Part A posed: given budget C, how much goes to model scale M and how much to data D? With the M ruler, the answer is a clean pair of power laws — drag the budget and watch the optimal split.
Fitting the curves the Chinchilla-style way (sweep ~10 model/data splits at each of 8 budgets, take the best of each), DeepSeek gets: Mopt ∝ C0.524 and Dopt ∝ C0.476. The exponents sum to 1 because C = M·D, and they're nearly even — model and data should grow almost in lockstep. Drag the slider:
Drag the compute budget C — the point rides the loss line; bars show the optimal M/D split
Slide the budget to see how model and data should grow together.
The point never leaves the line, and the two bars stay close. Because 0.524 and 0.476 are nearly equal, every 10× of compute means roughly 3.3× more model and 3.0× more data — almost a balanced split, not the lopsided one OpenAI predicted. Values illustrate the fitted trend.
Now the formula arrives — and it should feel like relief. It just writes the line you've been dragging in symbols:
Mopt=0.1715·C0.524 , Dopt=5.8316·C0.476
a + b = 0.524 + 0.476 = 1, exactly as C = M·D requires
With the honest ruler, DeepSeek's split (a ≈ 0.52) lands much closer to Chinchilla's balanced answer than to OpenAI's model-heavy one. The disagreement wasn't a mystery of datasets — it was largely an artifact of measuring model scale with the wrong ruler.
Eq. 4: Mopt = Mbase·Ca with Mbase = 0.1715, a = 0.5243; Dopt = Dbase·Cb with Dbase = 5.8316, b = 0.4757. Fitted via the IsoFLOP-profile method on 8 budgets from 1e17 to 3e20.
Step 6 · Better data bends the law
The cleaner your data, the more of each new budget should go to the model.
The split a ≈ 0.52 isn't a universal constant. While building DeepSeek, the team kept improving the dataset — and noticed the optimal split moved with data quality. This is the deepest finding in the paper.
Refit the same scaling law on three datasets of rising quality and watch the model exponent a climb while the data exponent b falls. Better data means each token is more informative, so it's worth spending relatively more of new compute on a bigger model and relatively less on yet more tokens:
early in-house data
a = 0.450 · b = 0.550
data-leaning
current in-house data
a = 0.524 · b = 0.476
near-balanced
OpenWebText2 (cleanest)
a = 0.578 · b = 0.422
model-leaning
This is the punchline. Two labs with different datasets will fit different optimal splits — not because scaling laws are unreliable, but because data quality itself is a hidden variable in the law. That likely explains the original OpenAI-vs-Chinchilla quarrel, and it means the “optimal split” you measure is also a quiet gauge of how good your data is.
Table 4: as quality rises (early → current in-house → OpenWebText2), a goes 0.450 → 0.524 → 0.578 and b goes 0.550 → 0.476 → 0.422. Intuition: high-quality data is more logically clear and easier to predict once learned, so extra budget is better spent on model capacity.
The payoff
Predict a 1000× run · ship the models · and check that you can explain it
Step 7 · The 1000× prediction comes true
Fit cheap runs, draw the line, and the giant 7B and 67B runs land right on it.
Honest ruler, pre-set dials, a balanced split that flexes with data quality. The whole point of all this was the one-shot bet from Step 1 — so does the prediction actually hold a thousandfold out?
DeepSeek fit the loss curve on small budgets, then extrapolated to predict the loss of the real 7B and 67B runs — about 1000× more compute than the fitting experiments. When the big runs finished, they landed essentially on the predicted line:
This closes the loop from Step 1. The one-shot, multi-million-GPU-hour run is no longer a gamble: you fit a line on cheap runs, read off the configuration, and the giant run lands where the line said it would. Prediction first, spending second.
Figure 5: the blue stars (7B, 67B) are well-predicted by the power-law fit on small models. Architecture: LLaMA-style Pre-Norm + RMSNorm + SwiGLU + RoPE; the 67B is 95 layers (deepened, not widened) and uses Grouped-Query Attention.
Step 8 · The scorecard & recap
The result: a 67B model that beats LLaMA-2 70B — and a chat model that tops GPT-3.5.
The methodology predicted the runs. So how good are the models it built — and have you actually got the ideas?
Report card · DeepSeek 67B vs. the field
GSM8K (grade-school math) · 67B base vs. LLaMA-2 70B
HumanEval (code) · 67B base vs. LLaMA-2 70B
MT-Bench (English chat) · 67B Chat DPO vs. GPT-3.5
AlignBench (Chinese chat) · 67B Chat DPO vs. GPT-3.5
prior status quo · “closed models lead on chat”
Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this paper.
Why did prior scaling-law papers disagree?
Largely because they measured “model scale” with raw parameter count, which mis-counts the real per-token work (it ignores attention's cost and over-counts the cheap vocabulary). That ruler error — up to 50% at small scale — shifted the fitted curves and pushed two careful labs to opposite splits.
What is non-embedding FLOPs/token, and why use it?
M is the floating-point work the Transformer body does to process one token — including attention, excluding vocabulary lookups. It reflects what the model actually computes, so the compute identity becomes the exact C = M·D, and the scaling curves fit cleanly.
How should you split a fixed compute budget?
Roughly evenly: Mopt ∝ C0.524 and Dopt ∝ C0.476. Each 10× of compute means about 3.3× more model and 3.0× more data — close to Chinchilla's balanced answer, not OpenAI's model-heavy one.
How does data quality change the answer?
Cleaner data raises the model exponent a (0.450 → 0.524 → 0.578 across three datasets) and lowers the data exponent b. Better data is more informative per token, so more of new compute should go to a bigger model — and this likely explains the original disagreement.
What does “longtermism” mean here, concretely?
Use cheap small-scale experiments to fit the laws, then predict the loss and ideal configuration of a run ~1000× larger before committing to it — turning a one-shot, multi-million-GPU-hour bet into a prediction that the 7B and 67B runs confirmed.
What happened next — this is the founding paper of the DeepSeek series. The same predict-then-spend discipline carried into DeepSeek-Coder, the Mixture-of-Experts line, and later reasoning models — each a bigger bet de-risked by the same scaling methodology.
“Instead of model parameters N, we adopt non-embedding FLOPs/token M to represent the model scale, leading to a more accurate optimal model/data scaling-up allocation strategy.” — DeepSeek-AI, 2024
The opening move of an open-weights challenger.
This paper did two things at once: it cleared up a genuine confusion in scaling laws, and it announced a lab that would build frontier-class open models on a disciplined, predictable recipe rather than a hunch.
A better ruler
Replacing parameter count with non-embedding FLOPs/token reconciled the OpenAI-vs-Chinchilla disagreement and tied the optimal split to data quality.
Predict, then spend
Fit cheap runs, forecast the loss of a 1000×-larger run, set hyperparameters in advance — turning a one-shot gamble into engineering.
The DeepSeek line
The founding paper of a series — DeepSeek-Coder, the MoE models, and later reasoning models — that pushed open weights toward the frontier.
Open-source LLMs got a defensible, reproducible basis for compute allocation — and a lab willing to publish the methodology rather than just the weights.