Most content-production workflows need voices and ambient audio that follow both detailed instructions (style, scene, effects) and, at times, a single reference clip for zero-shot cloning. This work addresses that dual need by treating data and model design together: expanding target coverage via curated captions and synthetic examples, then training a unified generator that handles multiple audio modalities and multi‑speaker scenarios.
Key Findings
- A purpose-built dataset and captioning pipeline (SwanData-Caption) increases coverage of speaker styles, acoustic scenes, and fine-grained controllable cues — so models can learn explicit instruction-to-audio mappings rather than relying on implicit heuristics.
- SwanVAE and a unified multimodal generator enable coherent synthesis across speech, music, and effects while preserving speaker identity in zero-shot settings — meaning a single model can produce multi-speaker dialogues and layered environmental audio.
- Reward-conditioned quality control, Engram conditioning, Unified MoE for multi-task modeling, plus curriculum training and GRPO post-training collectively raise expressiveness and robustness — leading to improved zero-shot and instruct metrics in evaluations.
- Empirical results report leading scores on several zero-shot/instruct benchmarks and higher human-rated expressiveness, indicating the approach scales to complex, multi-speaker instructive scenarios.
Who it's for and trade-offs
Great fit if you need an end-to-end approach to generate or prototype voices and scene audio for animation, games, podcasts, or advertising where either textual direction or a single reference clip should control output. It is less appropriate if you require open, fully reproducible baselines (the paper relies on substantial curated/synthetic data and multi-stage training) or extremely low-latency on-device inference without large models or specialist serving infrastructure.