Research map / Image generation / Fast diffusion sampling

IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling

ICML 2025 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IMPACT, which combines MAR-style iterative mask-based parallel decoding over continuous audio latents with a small MLP diffusion head and unconditional pre-training, achieving state-of-the-art FD/FAD on AudioCaps with lower latency than diffusion baselines.

Problem

Diffusion-based text-to-audio models (Tango, AudioLDM series) produce high fidelity audio but have high inference latency from slow diffusion sampling, while the discrete-token mask-based model MAGNET is fast but lags in audio quality.

Approach

Apply MAR-style iterative mask-based parallel decoding directly on continuous VAE audio latents, using a lightweight MLP diffusion head conditioned on a transformer latent encoder, plus an unconditional mask-modeling pre-training phase on unlabeled audio before text-conditional training.

Key results

• IMPACT base (5500h pre-train, 1200h text-cond, 64 decoding iterations) reaches FD 14.72 and FAD 1.13 on AudioCaps, versus EzAudio-XL FD 14.98 and Tango FAD 1.73.
• IMPACT base with 4 decoding iterations achieves FAD 2.85 and CLAP 0.321 versus MAGNET-S's FAD 3.22 and CLAP 0.287 at 50 decoding iterations.
• IMPACT base with 32 decoding iterations has latency 11.2 s for a batch of 8 on a V100, second only to MAGNET-S (6.9 s), and runs at 7.75 s for batch 64 vs MAGNET-S's 52.04 s.

Measured effects reported in the paper

Surprising findings

• A single-pass (non-iterative) variant performs far worse (FAD 12.26) than 32-iteration decoding (FAD 1.07), showing iterative decoding matters.
• Removing the CLAP text encoder leaves most metrics nearly unchanged, suggesting Flan-T5 embeddings alone suffice for text alignment.

Limitations

• Latency remains higher than MAGNET-S, the fastest baseline, at comparable batch sizes when using more decoding iterations.
• CLAP score lags Tango 2 and EzAudio, which are trained with DPO or data filtering.
• Random position selection replaces confidence-based selection since continuous latents have no confidence scores, and the paper provides no alternative analysis of its effect.

Experimental setup

Datasets: AudioSet AudioCaps WavCaps

Benchmarks: AudioCaps

Metrics: FD FAD KL IS CLAP REL OVL latency

Baselines: Tango Tango-full-ft Tango 2 AudioLDM2 EzAudio-L EzAudio-XL MAGNET-S MAGNET-L AudioGen v2 Make-an-Audio 2

Builds on: MAR MAGNET Latent diffusion models Classifier-free guidance AudioLDM

Closest papers by meaning

• Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.89)
• Masked Audio Generation using a Single Non-Autoregressive Transformer (ICLR 2024 · similarity 0.88)
• Fast Timing-Conditioned Latent Audio Diffusion (ICML 2024 · similarity 0.88)
• MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.87)
• ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.87)
• AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.86)