Most community recipes protected the recurrent Gated DeltaNet when applying aggressive 4-bit quantization to hybrid LLMs out of fear that recurrence would accumulate quantization noise. This paper overturns that assumption: applying NVFP4 W4A4 to every linear layer (496 total) of a Qwen3.8-27B hybrid model matches BF16 accuracy across multiple benchmarks while reducing memory and improving prefill throughput.
Key Findings
- Full 4-bit quantization can match BF16 within seed noise on perplexity (4K/32K) and downstream tasks (MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, RULER retrieval to 64K), so quantize-everything is a practical deployment recipe. This means smaller checkpoints and simpler serving stacks without substantial accuracy loss.
- Quantized model size drops to ~17.5 GiB and prefill latency improves by ~14–19%, so inference cost and memory pressure are materially reduced for long-context serving.
- The recurrent GDN gates are less sensitive than expected: nonlinearity parameterizations (softplus/exponential/sigmoid) compress GEMM error substantially, so gate errors translate to much smaller output errors.
- A four-part mechanism explains robustness: NVFP4 block scaling localizes outliers; gate parameterizations dampen GEMM noise; the delta-rule recurrence overwrites and forgets injected impulses within hundreds of tokens; and per-token quantization cost washes out with longer contexts.
Who it's for
Great fit if you operate or research long-context hybrid LLMs and need lower-memory, faster inference with minimal accuracy regression—teams running large models on single high-memory GPUs, or anyone building compact NVFP4 checkpoints for production. Look elsewhere if your deployment hardware or kernels do not support NVFP4-style FP4 formats or if you require an end-to-end QAT solution; this work is calibration-only post-training quantization (no QAT/distillation).
Mechanistic notes
The paper also documents a serving pitfall and practical fixes: per-module-calibrated NVFP4 weights can be mis-scaled by fused GEMM kernels; shipping harmonized KV-cache scales and corrected fused-kernel handling are part of the released recipe. The authors provide a calibrated NVFP4 checkpoint for immediate evaluation and serving.