AIAny. ← Back to AIAny

Lesson notes · One paper · the founding DeepSeek model

DeepSeek LLM: Scaling with Longtermism

By the end of this page you'll be able to explain why one quiet choice — how you measure a model's size — silently rewrites your whole budget plan, and how a lab uses cheap experiments to predict, in advance, the loss of a run that costs a thousand times more.

Subject LLM scaling laws · open weights Date Jan 5, 2024 Venue arXiv 2401.02954 · DeepSeek-AI Built 7B & 67B on 2T tokens

DeepSeek-AI — a large collaboration including Xiao Bi, Deli Chen, Damai Dai, Kai Dong, Daya Guo, Wenfeng Liang, Fuli Luo, Chong Ruan, Zhihong Shao, Dejian Yang, Chenggang Zhao, Qihao Zhu (authors listed alphabetically by last name).

The core idea

Measure the model's size correctly, and the scaling disagreement disappears.

Earlier scaling-law papers disagreed loudly on one practical question: given more compute, should you spend it on a bigger model or on more data? The disagreement was usually waved away as “different datasets.” DeepSeek pinned it to something concrete and fixable: everyone was measuring model size with raw parameter count, which quietly mis-counts the real cost of running the model.

Swap parameter count for non-embedding FLOPs per token — literally, the arithmetic the model does to read one token — and the conflicting results line up. The corrected law then says how to split a budget, lets you fix learning rate and batch size before a run, and predicts the loss of a giant run from cheap small ones. That is the “longtermism”: use small experiments to de-risk a single multi-million-GPU-hour bet.

A scaling law is only as good as the ruler it's drawn against. Fix the ruler, and a thousandfold extrapolation becomes a straight line you can trust.

PART A

The puzzle

You have a fixed pile of compute. Bigger model, or more data?

Step 1 · The black box & the one bet

A language model is a box you grade with one number — and you only get to train it once.

Here is the whole subject in one picture. A decoder-only Transformer reads text and predicts the next token (a sub-word chunk); do that across a corpus and you can score it. The score is the loss — how surprised the model is by the real next token. Lower is better. DeepSeek measures it as bits-per-byte on a held-out validation set, and that one number follows us through every step:

compute budget C 2T tokens of data your two levers DEEPSEEK LLM predicts P(next token) loss L bits-per-byte the catch: you train it once
Our running situation. A real lab spends millions of GPU-hours on one training run — so it must decide, before spending, how to split the budget. The whole paper is about predicting the loss L before the run.

And here is the bind that makes this a research problem at all. A frontier run is a single, enormous, one-shot bet. You cannot tune it by trial and error — there is no “try again.” So the real question isn't “what's the best model?” It's: can you predict the best configuration cheaply, in advance, and be right?

Key property — remember this one

One number, the loss in bits-per-byte, grades the whole model. Everything that follows is about predicting how low that number will go — before you commit the budget.

From the abstract: scaling laws in prior literature “present varying conclusions, which casts a dark cloud over scaling LLMs.” DeepSeek sets out to clear that cloud, then trains 7B and 67B base + chat models on a 2-trillion-token bilingual corpus.

Step 2 · Three quantities, one identity, one quarrel

Compute = model scale × data. The whole quarrel is what “model scale” means.

You've met the loss L and the one-shot bet. Now: what are you actually splitting? All of scaling rests on three quantities tied by a single identity.

Define each plainly — every law from here uses these three letters:

quantity 1 · compute

C

total training work

C — the compute budget. The total arithmetic you can afford. This is the thing held fixed; the question is how to spend it.

quantity 2 · model scale

M

how big the model is

M — model scale. How much model you build. Earlier work used parameter count N for this; DeepSeek's fix is to measure M differently — that's Step 3.

quantity 3 · data

D

⋯ more tokens ⋯

D — data, in tokens. How much text the model trains on. DeepSeek's final corpus is 2 trillion tokens, mostly English and Chinese.

C · compute budget M · model scale D · data (tokens)

spend your compute C on a model of scale M, run over D tokens — that's the identity everything balances on

The old shorthand was C ≈ 6·N·D — six floating-point operations per parameter per token. Convenient, but it bakes in raw parameter count N as the measure of model scale. And that is exactly where the field's disagreement was hiding. Look how far apart prior papers landed on the “bigger model vs. more data” split:

The quarrel, in two numbers

OpenAI (Kaplan 2020) said add compute mostly to the model: model exponent a ≈ 0.73, data exponent b ≈ 0.27. Chinchilla (Hoffmann 2022) said split it nearly evenly: a ≈ 0.49, b ≈ 0.51. Same goal, opposite advice — and billions of dollars of training ride on which is right.

Where Nopt ∝ Ca and Dopt ∝ Cb give the optimal model and data for a budget (a + b ≈ 1). The conflict between a ≈ 0.73 and a ≈ 0.49 is the “dark cloud” this paper sets out to clear (Table 4).

PART B

The fix

A better ruler for model size · set knobs in advance · ride a power law · and why data quality bends it

Step 3 · The better ruler — non-embedding FLOPs/token

Count the work, not the weights — and a 50% mismeasurement vanishes.

The quarrel hid inside the word “model scale.” So replace the loose ruler with a precise one — and do the arithmetic by hand on a small model to feel why it matters.

DeepSeek measures model scale as M, non-embedding FLOPs per token — the floating-point operations the Transformer body does to process one token, including attention but excluding the cheap vocabulary lookups. Compare three rulers for the same model. Take a small one with nlayer = 8, dmodel = 512, vocab 102,400, sequence length 4,096, and plug into the paper's formulas:

Ruler 6N₁ — non-embedding params only

72·nlayer·dmodel2. Counts the body's weights, ignores attention's own arithmetic.

72× 8× 5122= ≈ 151M

Ruler 6N₂ — complete params (adds vocabulary)

Adds 6·nvocab·dmodel. The vocabulary barely adds model capacity, yet inflates the count a lot at small scale.

151M+ 6 × 102,400 × 512= ≈ 466M

Ruler M — non-embedding FLOPs/token (the fix)

Adds attention's real cost 12·nlayer·dmodel·lseq, drops the vocabulary. This is what the model actually computes per token.

151M+ 12 × 8 × 512 × 4,096= ≈ 352M
Feel the gap

For this small model the “params” ruler 6N₁ reads 151M while the honest ruler M reads 352M — 6N₁ is only 0.43× of the truth, off by more than 50%. Use a ruler that's wrong by 50% and every point you fit the scaling curve to is shifted — which is precisely how two careful labs reached opposite conclusions.

Formulas (Eq. 2): 6N₁ = 72·nlayer·dmodel2; 6N₂ = 6N₁ + 6·nvocab·dmodel; M = 6N₁ + 12·nlayer·dmodel·lseq. The 6N₁/M ratio (Table 3) climbs from 0.43 at this scale toward 0.92 only at ~64B params — the error is worst exactly where the cheap experiments live.

Step 4 · Set the dials before you spend

Learning rate and batch size also follow power laws — so you can set them in advance.

With an honest ruler for M, you can fit clean curves. But a giant run also needs a learning rate and batch size — and you can't grid-search those at full scale. DeepSeek found you don't have to.

Across many small runs, the near-optimal batch size (how many tokens per update) and learning rate (how big each update step is) trace smooth power laws in the compute budget C. As C grows, the best batch size rises and the best learning rate falls — both predictably. So you read them off a fitted line instead of tuning by trial and error:

learning rate

ηopt = 0.3118 · C−0.1250

bigger budget → gentler steps

C →

Read η straight off the line for any budget.

batch size

Bopt = 0.2920 · C0.3271

bigger budget → larger batches

C →

Set B from the budget — no grid search.

a wide safe band

±0.25% loss

near-optimal is easy to hit

C →

A broad band stays within 0.25% of best.

Why this is freeing

The hyperparameters you'd normally babysit become a lookup. And the band of near-optimal settings is wide, so you don't need to hit them exactly — you just need to not be wildly off. The expensive run starts already well-tuned.

Eq. 1, fitted on compute budgets from 1e17 to 2e19, then validated at 1e20. A setting was counted “near-optimal” if its generalization error exceeded the minimum by no more than 0.25%.

Step 5 · The allocation law — the centerpiece

For each extra unit of compute, grow the model and the data by fixed, near-equal shares.

Honest ruler, dials pre-set. Now the question Part A posed: given budget C, how much goes to model scale M and how much to data D? With the M ruler, the answer is a clean pair of power laws — drag the budget and watch the optimal split.

Fitting the curves the Chinchilla-style way (sweep ~10 model/data splits at each of 8 budgets, take the best of each), DeepSeek gets: Mopt ∝ C0.524 and Dopt ∝ C0.476. The exponents sum to 1 because C = M·D, and they're nearly even — model and data should grow almost in lockstep. Drag the slider:

Drag the compute budget C — the point rides the loss line; bars show the optimal M/D split

10¹⁷ 10¹⁸ 10¹⁹ 10²⁰ 10²¹ 10²² 10²³ 10²⁴ compute budget C (FLOPs) — log scale validation loss — log
model scale Mopt
∝ C0.524
data Dopt
∝ C0.476
Model Mopt
∝ C0.524
growth
Data Dopt
∝ C0.476
growth
10¹⁷← drag → budget C10²⁴

Slide the budget to see how model and data should grow together.

The point never leaves the line, and the two bars stay close. Because 0.524 and 0.476 are nearly equal, every 10× of compute means roughly 3.3× more model and 3.0× more data — almost a balanced split, not the lopsided one OpenAI predicted. Values illustrate the fitted trend.

Now the formula arrives — and it should feel like relief. It just writes the line you've been dragging in symbols:

a + b = 0.524 + 0.476 = 1, exactly as C = M·D requires

The verdict on the quarrel

With the honest ruler, DeepSeek's split (a ≈ 0.52) lands much closer to Chinchilla's balanced answer than to OpenAI's model-heavy one. The disagreement wasn't a mystery of datasets — it was largely an artifact of measuring model scale with the wrong ruler.

Eq. 4: Mopt = Mbase·Ca with Mbase = 0.1715, a = 0.5243; Dopt = Dbase·Cb with Dbase = 5.8316, b = 0.4757. Fitted via the IsoFLOP-profile method on 8 budgets from 1e17 to 3e20.

Step 6 · Better data bends the law

The cleaner your data, the more of each new budget should go to the model.

The split a ≈ 0.52 isn't a universal constant. While building DeepSeek, the team kept improving the dataset — and noticed the optimal split moved with data quality. This is the deepest finding in the paper.

Refit the same scaling law on three datasets of rising quality and watch the model exponent a climb while the data exponent b falls. Better data means each token is more informative, so it's worth spending relatively more of new compute on a bigger model and relatively less on yet more tokens:

early in-house data

M 0.450 D 0.550

a = 0.450 · b = 0.550

data-leaning

current in-house data

M 0.524 D 0.476

a = 0.524 · b = 0.476

near-balanced

OpenWebText2 (cleanest)

M 0.578 D 0.422

a = 0.578 · b = 0.422

model-leaning

The resolution — tie it all together

This is the punchline. Two labs with different datasets will fit different optimal splits — not because scaling laws are unreliable, but because data quality itself is a hidden variable in the law. That likely explains the original OpenAI-vs-Chinchilla quarrel, and it means the “optimal split” you measure is also a quiet gauge of how good your data is.

Table 4: as quality rises (early → current in-house → OpenWebText2), a goes 0.450 → 0.524 → 0.578 and b goes 0.550 → 0.476 → 0.422. Intuition: high-quality data is more logically clear and easier to predict once learned, so extra budget is better spent on model capacity.

PART C

The payoff

Predict a 1000× run · ship the models · and check that you can explain it

Step 7 · The 1000× prediction comes true

Fit cheap runs, draw the line, and the giant 7B and 67B runs land right on it.

Honest ruler, pre-set dials, a balanced split that flexes with data quality. The whole point of all this was the one-shot bet from Step 1 — so does the prediction actually hold a thousandfold out?

DeepSeek fit the loss curve on small budgets, then extrapolated to predict the loss of the real 7B and 67B runs — about 1000× more compute than the fitting experiments. When the big runs finished, they landed essentially on the predicted line:

cheap fits 7B 67B compute C → (1000× span) loss →
Predicted ≈ delivered The blue stars are the actual 7B and 67B runs. They sit on the line drawn from experiments a thousand times cheaper. The cheap experiments de-risked the expensive bet — exactly the “longtermism” the title promises.
Why this is the whole point

This closes the loop from Step 1. The one-shot, multi-million-GPU-hour run is no longer a gamble: you fit a line on cheap runs, read off the configuration, and the giant run lands where the line said it would. Prediction first, spending second.

Figure 5: the blue stars (7B, 67B) are well-predicted by the power-law fit on small models. Architecture: LLaMA-style Pre-Norm + RMSNorm + SwiGLU + RoPE; the 67B is 95 layers (deepened, not widened) and uses Grouped-Query Attention.

Step 8 · The scorecard & recap

The result: a 67B model that beats LLaMA-2 70B — and a chat model that tops GPT-3.5.

The methodology predicted the runs. So how good are the models it built — and have you actually got the ideas?

Report card · DeepSeek 67B vs. the field

GSM8K (grade-school math) · 67B base vs. LLaMA-2 70B

63.4 vs 58.4

HumanEval (code) · 67B base vs. LLaMA-2 70B

42.7 vs 28.7

MT-Bench (English chat) · 67B Chat DPO vs. GPT-3.5

8.76 vs 8.39

AlignBench (Chinese chat) · 67B Chat DPO vs. GPT-3.5

6.69 vs 6.08

prior status quo · “closed models lead on chat”

challenged, open
Code, math, reasoning 67B base beats LLaMA-2 70B clearly on HumanEval, MBPP, GSM8K, MATH and BBH.
Open-ended chat after SFT + DPO, 67B Chat tops GPT-3.5 on both MT-Bench (English) and AlignBench (Chinese).
Scale shows on hard tasks on held-out LeetCode and the Hungarian exam the 67B far outruns small models that look fine on standard benchmarks.
Honest practice no benchmark decoration; they excluded multiple-choice data to avoid overfitting to leaderboards.

Now you can explain it. Five questions — answer each out loud before opening it. If all five come easily, you've genuinely got this paper.

Why did prior scaling-law papers disagree?

Largely because they measured “model scale” with raw parameter count, which mis-counts the real per-token work (it ignores attention's cost and over-counts the cheap vocabulary). That ruler error — up to 50% at small scale — shifted the fitted curves and pushed two careful labs to opposite splits.

What is non-embedding FLOPs/token, and why use it?

M is the floating-point work the Transformer body does to process one token — including attention, excluding vocabulary lookups. It reflects what the model actually computes, so the compute identity becomes the exact C = M·D, and the scaling curves fit cleanly.

How should you split a fixed compute budget?

Roughly evenly: Mopt ∝ C0.524 and Dopt ∝ C0.476. Each 10× of compute means about 3.3× more model and 3.0× more data — close to Chinchilla's balanced answer, not OpenAI's model-heavy one.

How does data quality change the answer?

Cleaner data raises the model exponent a (0.450 → 0.524 → 0.578 across three datasets) and lowers the data exponent b. Better data is more informative per token, so more of new compute should go to a bigger model — and this likely explains the original disagreement.

What does “longtermism” mean here, concretely?

Use cheap small-scale experiments to fit the laws, then predict the loss and ideal configuration of a run ~1000× larger before committing to it — turning a one-shot, multi-million-GPU-hour bet into a prediction that the 7B and 67B runs confirmed.

What happened next — this is the founding paper of the DeepSeek series. The same predict-then-spend discipline carried into DeepSeek-Coder, the Mixture-of-Experts line, and later reasoning models — each a bigger bet de-risked by the same scaling methodology.

“Instead of model parameters N, we adopt non-embedding FLOPs/token M to represent the model scale, leading to a more accurate optimal model/data scaling-up allocation strategy.” — DeepSeek-AI, 2024
The profound impact

The opening move of an open-weights challenger.

This paper did two things at once: it cleared up a genuine confusion in scaling laws, and it announced a lab that would build frontier-class open models on a disciplined, predictable recipe rather than a hunch.

2024 · the method

A better ruler

Replacing parameter count with non-embedding FLOPs/token reconciled the OpenAI-vs-Chinchilla disagreement and tied the optimal split to data quality.

labs · the practice

Predict, then spend

Fit cheap runs, forecast the loss of a 1000×-larger run, set hyperparameters in advance — turning a one-shot gamble into engineering.

2024 → present · the series

The DeepSeek line

The founding paper of a series — DeepSeek-Coder, the MoE models, and later reasoning models — that pushed open weights toward the frontier.

Open-source LLMs got a defensible, reproducible basis for compute allocation — and a lab willing to publish the methodology rather than just the weights.