Research map / Image generation / Autoregressive and latent generation
Masked Audio Generation using a Single Non-Autoregressive Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MAGNeT, a single non-autoregressive masked-span transformer over EnCodec streams for text-to-music and text-to-audio generation, with rescoring by an external AR model and a hybrid AR/NAR variant.
Problem
Autoregressive audio language models have high latency for interactive music/audio generation, while diffusion models need many steps and struggle with long-form audio.
Approach
A single non-autoregressive masked generative transformer over multi-stream EnCodec tokens, with span masking, restricted attention context for codebooks above the first, external-model rescoring of spans during iterative decoding, CFG annealing, and a hybrid training scheme that allows AR prompt then parallel decoding.
Key results
- MAGNeT-small reaches FAD 3.3 and KL 1.123 on MusicCaps versus MusicGen-large FAD 3.4 and KL 1.23, with latency 4.0s vs 41.3s.
- MAGNeT reports up to about 7x faster inference than the autoregressive MusicGen baseline and latency as low as 600 ms for a single sample at batch size 2 (10-second generations, A100).
- Hybrid-MAGNeT with a 1-second AR prompt outperforms the full autoregressive baseline on FAD for 30-second generations, with latency 3.2s vs 17.6s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add restricted temporal context for codebooks above the first (ablation) | MAGNeT · In-domain test set, span length 3 | FAD: 0.8221 -> 0.61 | Helped |
| Use 60ms spans (span length 3) instead of other tested span lengths (ablation) | MAGNeT · In-domain test set | FAD: 0.61 | Helped |
| Add model rescoring (ablation) | MAGNeT-small · Music generation | FAD: 3.7 -> 3.3 | Helped |
| Add model rescoring (ablation) | MAGNeT-large · Music generation | overall quality: 82.19±1.23 -> 84.43±1.24 | Helped |
| Add model rescoring (ablation) | MAGNeT · Music generation | latency: 2.5 -> 4.0 | Hurt |
| Anneal CFG from 10 to 1 instead of keeping it constant at 10 (ablation) | MAGNeT · In-domain test set | FAD: 0.629 -> 0.61 | Helped |
| Reduce higher-codebook decoding to 1 step per level instead of 10 (ablation) | MAGNeT · In-domain test set | FAD: 8% increase of FAD compared to 10 steps per higher levels | Hurt |
| Increase the autoregressive prompt duration in the hybrid model (ablation) | Hybrid-MAGNeT · 30-second generation | FAD | Helped |
| Use a 1-second autoregressive prompt before non-autoregressive decoding (ablation) | Hybrid-MAGNeT · 30-second generation | FAD: 3.2s of latency down from 17.6s | Helped |
| Reduce first-codebook decoding steps from 20 to 10 (ablation) | MAGNeT · In-domain test set | FAD: 0.61 -> 0.65 | Hurt |
Surprising findings
- Restricting self-attention for codebooks above the first to a ~200ms neighborhood consistently improved FAD across span lengths.
- Reducing decoding steps for higher codebook levels hurts quality less than reducing steps for the first level.
Limitations
- The non-autoregressive model re-encodes the whole sequence at every decoding step, with no KV caching, so throughput is bounded at large batch sizes.
- Text-to-music FAD is worse than AudioLDM2 and CLAP is lower than some baselines; quality is comparable rather than superior.
- Rescoring adds inference cost and depends on an external pre-trained model (MusicGen/AudioGen).
Experimental setup
Datasets: ShutterStock music Pond5 music MusicCaps AudioCaps AudioSet
Benchmarks: MusicCaps AudioCaps
Metrics: FAD KL CLAP score overall quality (human MOS) relevance (human MOS) latency throughput
Baselines: MusicGen AudioLDM 2 MusicLM Mousai AudioGen DiffSound Make-an-Audio
Builds on: MaskGIT Mask-Predict MusicGen EnCodec classifier-free guidance
Closest papers by meaning
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
- IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling (ICML 2025 · similarity 0.88)
- MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer (ICLR 2025 · similarity 0.88)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.87)
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.86)
- ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.