Re-derives LLM scaling laws, tracing prior disagreements to how compute budget was modeled, then trains 7B and 67B models on 2T tokens. The 67B model beats LLaMA-2 70B on code, math, and reasoning; its chat variant tops GPT-3.5 on open-ended evals.

Reworks the MoE layer to push each expert toward a narrow specialty: split experts into many finer ones and activate more per token, plus reserve a few always-on shared experts for common knowledge. A 2B model matches GShard 2.9B; at 16B it rivals LLaMA2 7B on ~40% of the compute.

A family of open code models (1.3B-33B) trained from scratch on 2T tokens of project-level code, using a 16K-window fill-in-the-blank objective. Beats Codex and GPT-3.5 on code benchmarks and ships under a license permitting commercial use.

Reaches 51.7% on the competition-level MATH benchmark with a 7B model and no tools or voting, rivaling Gemini-Ultra and GPT-4. Built on a 120B-token math corpus mined from Common Crawl, and introduces GRPO, a memory-efficient PPO variant for reasoning.

A 236B-parameter Mixture-of-Experts LLM that activates only 21B parameters per token, cutting training cost 42.5% versus a dense 67B model and shrinking the KV cache 93.3% via Multi-head Latent Attention, with 128K context.

Open-weight Mixture-of-Experts LLM with 671B total parameters but 37B activated per token, trained on 14.8T tokens for 2.788M H800 GPU-hours. Matches leading closed models at a fraction of typical training cost via FP8 and architectural tricks.

Shows that LLM reasoning can be incentivized through pure reinforcement learning, with no human-annotated reasoning traces. Self-reflection, verification, and strategy-switching emerge on their own, and the patterns transfer to distill smaller models.

Preview of an MoE model family (V4-Pro: 1.6T params, 49B active; V4-Flash: 284B, 13B active) built for 1M-token contexts. A hybrid attention design cuts single-token inference FLOPs to 27% and KV cache to 10% versus V3.2 at million-token length.