Most text generation still treats language as discrete tokens even though images, audio and video increasingly use continuous latents. That mismatch matters: tokenized decoders force compromises in latent design or require lossy compression to fit diffusion models. AURORA-LM flips the trade-off—keep a full-width, decodable latent and adapt the generative model to it—so diffusion can operate over continuous text representations without shrinking decoder-facing capacity.
Key Findings
- Preserves a high-capacity, prefix-aligned latent: a Query-based encoder–decoder builds latents that remain fully decodable by a standard decoder, avoiding the fidelity loss of heavily compressed latents. This means generated latents can map back to token-level text without sacrificing representational width.
- Block-causal Diffusion Transformer with flow matching: generates left-to-right at block granularity while denoising positions inside a block in parallel, enabling faster sampling while keeping autoregressive structure across blocks.
- Training innovations for stability and fidelity: restricts only the noisy-input pathway (retain clean-latent prediction target), calibrates noise-level distribution to latent width, and introduces self-trajectory consistency to align independently sampled training noise with iterative denoising at inference.
- Empirical gains: reports strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization; scaling experiments (≈1B parameters, ~1500 EFLOPs) further improve results and surpass a larger publicly released latent-diffusion language model under matched evaluation.
Who it's for and trade-offs
Great fit if you are researching alternative text-generation formats (continuous-latent/diffusion) or building generative models that need richer decodable latents than token embeddings allow. The approach is most relevant for offline research and high-compute model development: the paper’s experiments use substantial compute and Ascend NPUs, and block-causal diffusion adds modeling complexity compared to standard autoregressive LMs. Look elsewhere if you need low-latency, lightweight production text generation, or strict token-level determinism with minimal inference steps.
Where it fits
AURORA-LM positions continuous-latent diffusion as a viable path for language generation, bridging the gap between diffusion paradigms common in images and discrete-token LMs. It sits alongside other latent-diffusion language efforts but emphasizes retaining decoder-facing capacity instead of compressing latents to ease modeling.
Method highlights
- Architecture: Query-based encoder–decoder to produce prefix-aligned, decodable latents; a Block-causal Diffusion Transformer for generative modeling.
- Optimization: flow-matching objective adapted to latent diffusion; noise scheduling calibrated to latent dimensionality; self-trajectory consistency to reduce train/inference mismatch.
These design choices target better fidelity to token-level text while enabling continuous-latent diffusion modeling. The paper documents empirical trade-offs and scaling behavior that help decide whether continuous-latent diffusion is a practical alternative for a given research or engineering goal.