Why this matters Looped (recurrent) Mixture-of-Experts (MoE) models let a shared block be applied multiple times to trade inference compute for accuracy, but not all loop visits contribute equally. LatentLoop shows that intentionally treating the first and second visits differently—compressing the first visit's sequence and reallocating compute to capacity—can improve likelihood and downstream scores under matched budgets.
Key Findings
- Empirical asymmetry: learned residual coefficients reveal a smaller update scale on the first visit; reducing only the first-visit scale preserves or improves performance while reducing the second-visit scale hurts it.
- LatentLoop method: compresses the sequence for the first loop visit, retains full resolution for the second, shares MoE expert weights across visits but uses separate attention parameters, and reallocates saved FLOPs to increase hidden size.
- Quantitative results: evaluated across 15 configurations spanning 1.3B–7B parameters (largest budget > 1e21 FLOPs). At a fitted NLL of 1.988, LatentLoop predicts 19.0% compute savings versus vanilla MoE and 7.9% versus a looped baseline, and it improves average scores on nine benchmarks in 0-shot and 5-shot settings.
Who it's for and tradeoffs
Great fit if you research model architectures that combine sparse routing with iterative/looped computation, and if you want to optimize accuracy-per-FLOP for MoE LLMs at the 1–7B parameter scale. Look elsewhere if you need off-the-shelf inference simplicity: asymmetric sequence resolutions require separate attention parameters and careful engineering for dispatch/communication, and the paper's experiments focus on up to 7B-parameter models (hardware and software implications may vary at larger scales).
Where it fits
LatentLoop sits at the intersection of looped/recurrent architectures and sparse Mixture-of-Experts designs. It complements prior looped MoE work by explicitly reallocating compute across visits rather than treating loop iterations symmetrically, offering a practical axis for FLOP-constrained model scaling.