DeepSeekMoE

Step 1 promised that a token pays for 2 experts while 16 sit on the shelf. The device that makes the choice — the router — is small enough to compute by hand, so let's do exactly that.

Here is the machine as it existed before this paper: the GShard-style MoE layer of 2021, the baseline DeepSeekMoE starts from. Enzyme arrives at the layer as a hidden vector u (the running summary of the token after attention). Every expert i owns a learned vector ei called its centroid — think of it as the expert's business card, pointing at the region of meaning-space it claims. Routing is three cheap moves:

Dot, squash, keep two. Take the dot product u·ei with all 16 centroids (how aligned is this token with each expert's specialty?), push the 16 scores through a softmax (exponentiate and normalize, so they become positive and sum to 1), and keep only the top 2. Those two scores become gate values — mixing weights — and everyone else's gate is 0, which is precisely why 14 experts cost nothing. In the figure below, follow enzyme's vector up through the router: the two teal boxes are the winners, and their outputs are blended by the gates before rejoining the residual stream.

That one line is the whole layer: the three moves you just did on the dial, written compactly. Since only K gates are nonzero, only K experts ever compute — sparsity is enforced by the gate, not by any special hardware trick.

You just routed enzyme to 2 winners out of 16. Now ask the question the paper asks: after training on trillions of tokens, what does each of those 16 experts end up containing?

Run the numbers from the router's side. With top-2 over 16, each expert receives on average an eighth of the entire corpus — enzyme kinetics today, Python tomorrow, sonnets on Thursday. All of it must be packed into one set of FFN weights and, worse, those weights all fire together whenever the expert is chosen. The paper calls this knowledge hybridity: the left pane below is one expert's crowded shelf, and the token chips crammed into it have nothing to do with each other.

The second leak is subtler. Every token — biological or not — needs the same base skills: English glue words, agreement, spelling. Whichever experts a token lands on must supply those skills, so many experts each learn a private copy. That's knowledge redundancy — the right pane, the same glue chip duplicated shelf after shelf, parameters spent on repetition instead of new knowledge.

Step 3 blamed both leaks on coarse-grained choice. The first edit attacks the coarseness directly — while touching neither the parameter count nor the compute bill.

The move: slice each expert FFN into m smaller ones by cutting its intermediate hidden width to 1/m of the original. With m = 4, the 16 experts become 64 quarter-size experts. Each is a quarter as big and a quarter as expensive — so, to spend the same compute, the router now activates 4 × 2 = 8 of them instead of 2. Enzyme no longer gets two generalists; it gets an eight-member committee of narrow specialists.

Honest admission: this was the part of the paper we re-read the most. "More experts, more activations, same cost" sounds like an accounting trick — surely something got bigger? The way out is to write the two ledger lines yourself and watch both stay fixed while a third quantity moves:

The figure makes the same ledger visual. Both strips have the same total width — that's the stored parameters — and the same amount of teal — that's the per-token compute. The only difference is how finely the width is diced, and therefore how precisely the teal can be placed.

Why does finer dicing fight hybridity? Because knowledge can now be decomposed. The old expert 3 held enzyme kinetics, cell membranes and Greek roots in one lump that always fired together; after the cut, those can live in different quarter-experts, and the router summons only the ones this token needs. Each small expert keeps a tighter, cleaner portfolio — the paper's phrase is that knowledge is "decomposed more finely and learned more precisely."

Step 4's ledger showed nothing got bigger. So what exactly did the edit buy? Teams. Let's count them — this is the number the whole paper leans on.

Start with the old layer, and count by hand. A "team" is a set of experts serving one token. Choosing 2 out of 16: the first pick has 16 options, the second 15, and order doesn't matter, so 16 × 15 / 2 = 120 possible teams. That's the entire menu — every token in every context must be served by one of 120 combinations.

Now the fine-grained layer: choosing 8 out of 64. Same counting rule, eight picks deep — written as the binomial C(64, 8) = 64! / (8! · 56!) — and it comes to 4,426,165,368. From 120 to 4.4 billion, a 37-million-fold richer menu, and the ledger from Step 4 says we paid nothing for it. This is the paper's own worked example, and it's the sense in which segmentation attacks hybridity: with a menu that rich, the router can compose a bespoke committee per token instead of forcing tokens into 120 pigeonholes.

Two honest boundaries before we move on. First, 4.4 billion is a count of possible routing patterns, not knowledge the model has already earned — the router still has to learn to use the menu, and Step 7 adds guardrails to keep it from collapsing onto a few favorites. Second, m can't grow forever: matrix multiplies below a certain size waste GPU throughput, which is why the 16B model stops at quarters and only the 145B trial dares eighths (§5.1.2).

Finer slicing lets each small expert stay narrow — that was leak 1. But leak 2 is still open: all 64 of them would still each learn a private copy of the glue that every sentence needs.

The second edit is almost embarrassingly direct: take Ks of the experts — one, in the 2B model — and pin it on. Every token visits the shared expert, deterministically, no router vote at all. To keep the compute bill pinned too, the router now picks only 8 − 1 = 7 routed experts from the remaining 63. In the slot picture below, enzyme's eight activation slots are one amber pin plus seven teal choices.

Why does one pinned expert fix redundancy? Because it changes what the routed experts are for. Common knowledge — the glue in "The … speeds up …" — now has a guaranteed home that every token passes through, so a routed expert gains nothing by storing its own copy. Gradient descent quietly stops paying for duplicates, and the 63 routed experts compress toward what is genuinely theirs. The paper's word for the effect is isolation: common knowledge is isolated out of the routed pool.

There is a price, and the paper doesn't hide it: pinning a slot shrinks the menu. Choosing 7 from 63 gives C(63,7) = 553,270,671 teams — half a billion, well under Step 5's 4.4 billion. The bet is that a permanent desk for the basics is worth more than those combinations. Is it? Make your own call before revealing the measurement:

Both edits are in place. Time to reassemble the full layer that enzyme actually traverses in DeepSeekMoE 2B — Step 2's diagram, upgraded.

Read the figure bottom-up, exactly as before: u rises into the router, which now scores only the 63 routed experts and keeps 7. Meanwhile the amber path on the left bypasses the router entirely — the shared expert runs unconditionally, with no gate. Everything meets at the same summation: seven gated teal outputs, one ungated amber output, plus u itself.

One loose end from Step 5: with 63 experts to choose from, what stops the router from feeding its early favorites forever — the rich-get-richer spiral known as routing collapse? Training adds two gentle penalty terms. An expert-level balance loss nudges the 63 routed experts toward comparable usage; its coefficient is kept deliberately tiny (0.01 at 2B, 0.001 at 16B) because forcing perfect balance would override the router's actual judgment. A device-level balance loss ignores individual experts and only evens out the compute landing on each GPU — the 145B run sets it 17× stronger than the expert-level one, balancing machines, not experts.

The design case is complete — two edits, zero extra budget. Now the burden of proof: do the edits buy measurable quality when nothing else is allowed to vary?

The validation setup is as controlled as it gets. Five models, the same 2.0B total parameters for every MoE (0.2–0.3B activated), the same 100B training tokens, the same everything else. The lineup below is a short history of MoE routing: a dense baseline sized like the activated compute, Hash Layer (top-1 by hash), Switch Transformer (top-1 learned), GShard (top-2 learned — Step 2's machine), and DeepSeekMoE (1 shared + 7 of 63). Lower loss is better; the bars tell the ranking at a glance.

Beating peers is one thing; the paper also asks the sharper question from Step 3 — how much of the leaked potential did we recover? Two harder yardsticks. First, a GShard scaled to 1.5× the expert parameters and compute: DeepSeekMoE ties it (1.808 vs 1.808) from a smaller machine. Second, the ceiling: a model with the same 16 units of expert parameters where all 16 experts run for every token — no routing, no wrong choices, every parameter available always. That costs 5.7× the FLOPs and defines the best any router could hope to reach with this parameter pool. Its loss: 1.806. DeepSeekMoE sits at 1.808 — 0.002 from the ceiling — while paying the sparse price. In the table, read the FLOPs column against the loss column.

Step 8 says the model is better. It doesn't yet say why — maybe something other than specialization improved. So the paper operates on the trained 2B model three times and watches what breaks.

Probe 1 — remove the favorites. For each token, mask the routed experts the router most wanted, and make it route among the rest. If experts were interchangeable generalists, understudies would cover fine. The left chart shows the result: DeepSeekMoE's loss (cinnabar) climbs far more steeply than GShard×1.5's (gray) — its experts are not interchangeable. The paper reads the steeper curve as lower redundancy: each routed expert holds something the others genuinely don't.

Probe 2 — use fewer experts. Run the trained model with only 4 routed picks instead of 7. On the right chart, the cinnabar curve crosses the gray dashed line at 4 experts: with roughly five-eighths of the activated expert parameters, DeepSeekMoE already matches full GShard. Knowledge sits where the router expects it, so fewer, sharper picks suffice.

Probe 3 — halve it and retrain. The strongest version: train a fresh model from scratch with 1 shared + only 3 of 63 routed activated — half of GShard's activated expert parameters. It still beats GShard across HellaSwag, PIQA, ARC, TriviaQA and NaturalQuestions (Figure 6). Not a surgical trick on a lucky checkpoint; the efficiency is in the architecture.

Everything so far is a 2B model fed 100B tokens — a lab bench. The fair worry: does the specialization story survive a real training run?

DeepSeekMoE 16B keeps the recipe (2 shared + 6 of 64 quarter-experts per layer) and trains on 2T tokens — the same diet as LLaMA2 7B. The scoreboard below is the head-to-head: look first at the FLOPs row, then at everything under it. Wins on most rows, a standout gap on code and math (the pretraining corpus is rich in both), all for 39.6% of the compute.

The MMLU row is the honest asterisk. On multiple-choice suites the 16B model trails slightly, and the paper's own diagnosis is instructive: with FFNs replaced by cheap sparse experts, only ~0.5B of its activated parameters are attention (a dense 7B has ~2.5B), and multiple-choice performance tracks attention capacity. The weakness isn't the expert layer misfiring — it's the corner that didn't get more parameters. A successor architecture (DeepSeek-V2's MLA) is largely about fixing that corner.

Before the bookend, close the loop yourself. Five questions, one per load-bearing idea — each opens onto a two-sentence answer.