AIAny
Large Language Model Papers2026

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Preview of an MoE model family (V4-Pro: 1.6T params, 49B active; V4-Flash: 284B, 13B active) built for 1M-token contexts. A hybrid attention design cuts single-token inference FLOPs to 27% and KV cache to 10% versus V3.2 at million-token length.

Visual ExplainerVisit Website

Introduction

Long-context inference has been gated by attention's quadratic cost, not by model quality. DeepSeek-V4 attacks the cost side directly: at a one-million-token context, V4-Pro spends only 27% of the single-token FLOPs and 10% of the KV cache of V3.2, while V4-Flash drops to 10% of FLOPs and 7% of cache. The point is to make million-token contexts a routine operating mode rather than a stunt, opening room for longer test-time scaling and long-horizon agentic work.

Key Findings
  • The efficiency comes from a hybrid attention scheme: Compressed Sparse Attention (CSA) compresses KV along the sequence dimension then applies sparse attention, while Heavily Compressed Attention (HCA) compresses KV harder but keeps attention dense — so cost stays bounded as context grows.
  • Manifold-Constrained Hyper-Connections (mHC) constrain the residual mapping to doubly stochastic matrices (spectral norm ≤ 1), fixing the numerical instability that broke earlier hyper-connection stacks at depth.
  • Training uses the Muon optimizer for faster convergence, FP4 quantization-aware training for MoE expert weights, and a two-stage post-training pipeline: independent domain specialists (math, code, agent) consolidated into one model via on-policy distillation.
  • On internal evals V4-Pro-Max outperforms Claude Sonnet 4.5 and approaches Opus 4.5 on agentic coding, and surpasses Gemini-3.1-Pro on long-context academic benchmarks; it still trails frontier closed models on knowledge by roughly 3–6 months.
Methodology

The series keeps the DeepSeekMoE and Multi-Token Prediction stack from V3, swapping the routing affinity activation to Softplus, removing the routing-target-node cap, and replacing dense FFNs in early blocks with hash-routed MoE layers. Both models pre-train on 32T+ tokens (V4-Pro on 33T) with native 1M-context support.

Great Fit If / Look Elsewhere

Worth studying if you care about the architecture of efficient ultra-long context, MoE routing, or RL-based post-training at frontier scale. Look elsewhere if you want a turnkey small model: this is a preview report on trillion-parameter systems, and the authors note the design stays deliberately complex with several empirically-validated-but-not-fully-understood tricks.

Back

Information

Categories

Tags

More Items

Reinforcement Learning Papers2026
Icon for item

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu +18

Enables RL post-training with million-token prompts under a fixed GPU budget by evaluating shared prompt state without autograd, retaining only minimal model state, and replaying short response branches; instantiated as GRPO and demonstrated on Qwen3.6-27B and GLM-5.2 up to multi-million token execution.

RLllmqwenmLOpsai-train+1
Reinforcement Learning Papers2026
Icon for item

Weak-to-Strong Generalization via Direct On-Policy Distillation

Shiyuan Feng, Huan-ang Gao +8

Transfers RL-induced policy shifts from a smaller 'weak' teacher to a stronger target by using the teacher's post-/pre-RL log-ratio as a dense implicit reward applied on the student's on-policy states. Enables reuse of RL supervision without running RL rollouts on the target, improving sample/time efficiency.

RLLLMreasoningqwenpaper+2
Large Language Model Papers2026
Icon for item

KronQ: LLM Quantization via Kronecker-Factored Hessian

Donghyun Lee, Yuhang Li +2

Introduces KronQ, a post-training quantization framework that incorporates gradient covariance via a Kronecker‑factored Hessian to guide input/output weight rotations and sensitivity-driven mixed-precision allocation. Demonstrates stable 2-bit weight-only quantization on LLaMA-3-70B (7.93 PPL).

paperllmLLMtransformerspytorch+1