The headline number is the one that breaks intuition: a 671B-parameter model trained for roughly $5.6M in compute. DeepSeek-V3 makes the case that the next frontier isn't more GPUs but better systems engineering — co-designing the model, the training algorithm, and the FP8 numerical stack so a Mixture-of-Experts model activates just 37B of its 671B parameters per token without the load-balancing headaches that usually wreck sparse training.
Key Findings
- An auxiliary-loss-free load-balancing scheme replaces the usual balancing loss, so experts stay evenly used without the accuracy tax that auxiliary losses impose.
- Multi-head Latent Attention (MLA) compresses the KV cache, cutting the inference memory that normally makes long-context serving expensive.
- A multi-token prediction objective densifies the training signal and doubles as speculative decoding at inference time.
- Native FP8 mixed-precision training across 2.788M H800 GPU-hours, with reported stability and zero loss-spike rollbacks across the entire 14.8T-token run.
How It Holds Up
On knowledge, math, and code benchmarks it lands ahead of other open-weight models and within reach of GPT-4o and Claude-3.5-Sonnet — a gap that, before this paper, most assumed required a closed lab's budget. The recipe, not just the weights, is the contribution.
Who It's For
Great fit if you're a researcher or infra team studying efficient large-scale training, or want strong open weights you can self-host and fine-tune. Look elsewhere if you need a turnkey hosted API with guarantees, or lack the multi-node H800-class hardware to actually run a 671B MoE — the open weights don't make the serving footprint small.