Most scaling work adds dense capacity; K2 Horizon takes a different path by introducing sparsity into attention as well as FFNs. The key insight is that routing in attention (MoVA) plus MoE feed‑forwards lets the model deliver near–dense-32B performance while only activating a few billion parameters per token, enabling much larger effective capacity for the same runtime cost.
Key Capabilities
- High capability per active parameter: stores ~36B parameters but activates ~4–4.7B core parameters per token, which reduces per-token compute while preserving high reasoning and agentic performance compared with many larger dense models — so what: lower inference cost for similar task performance.
- Very long context: native 524,288-token context from midtraining onward — so what: supports retrieval- or document-heavy workflows without external chunking or retrieval shims.
- Open, reproducible release: final weights, planned intermediate checkpoints, training data recipes, and training code are published under an open license — so what: researchers can study capability evolution, reproduce training stages, and adapt or fine-tune from known checkpoints.
- Agent and serving support: validated with vLLM, SGLang and common Transformer tooling and recommended parser/settings for agent use — so what: practical to deploy in agentic and tool-using pipelines with validated recipes.
Who It's For & Tradeoffs
Great fit if you need a model that balances high capability with lower per-token activation cost (researchers building long-context agents, groups experimenting with sparse attention, and teams wanting an open-weight frontier model). Look elsewhere if you need the simplest possible deployment (dense models often have fewer runtime implementation constraints) or if you require the absolute top scores on every dense-model benchmark — MoVA introduces engineering complexity (serving, router numerics, expert parallelism) and uncommon runtime requirements that increase deployment friction.
Where It Fits
K2-Horizon-MoVA-36B-A4B sits between dense ~30–32B models and much larger MoE fleets: it targets users who want near–large-model performance but prefer lower active-compute and full transparency (open weights + data + code). It's particularly suited to long-document reasoning, agentic tool use, and experiments into sparse-attention architectures.
How It Works (brief)
MoVA (Mixture-of-Value Attention) routes value vectors across attention heads so different attention experts handle different parts of the context; combined with MoE FFNs this creates capacity that is sparse at inference time. The published model includes architecture metadata (48 layers, 2,560 hidden size, large expert pool and routing counts) and evaluation tables showing benchmark tradeoffs versus other open and closed models.
