Research map / Image generation / Text-to-image diffusion
AudioGenX: Explainability on Text-to-Audio Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AudioGenX, a perturbation-based explainer that learns soft token-importance masks for text-to-audio models (AudioGen) using factual and counterfactual cosine-similarity objectives in cross-attention.
Problem
Text-to-audio generation models lack transparency about how each input text token influences the generated audio, and existing transformer explanation methods (attention rollout, Grad-CAM, AtMan, Chefer et al.) are poorly suited to cross-attention in audio token generation.
Approach
Train a small Explainer network to output soft masks over text tokens that is applied inside cross-attention; the mask is optimized with a factual loss (keep latent audio-token representation similar) and a counterfactual loss (make it dissimilar when important tokens are removed), plus L1/L2 regularization, and per-audio-token masks are averaged to explain whole audio or chosen intervals.
Key results
- AudioGenX achieves Fid_F 0.132 (lower better) and Fid_CF 0.405 (higher better) vs. best baseline Grad-CAM-e Fid_CF 0.235 and AtMan Fid_F 0.195, with mask size 0.455.
- KL_CF of 4.259 for AudioGenX vs. 2.457 for Grad-CAM-e, the strongest baseline.
- Editing with AudioGenX importance re-weighting reduces FAD from 16.85 to 2.68 on 30 failure cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine factual and counterfactual losses instead of using either loss alone (ablation) | AudioGenX · AudioCaps test | Fid_CF: 0.360 -> 0.405; 0.385 -> 0.405 | Helped |
| combine factual and counterfactual losses instead of using either loss alone (ablation) | AudioGenX · AudioCaps test | KL_CF: 3.658 -> 4.259; 3.977 -> 4.259 | Helped |
| combine factual and counterfactual losses instead of using either loss alone (ablation) | AudioGenX · AudioCaps test | Fid_F: 0.145 ± 0.004 -> 0.132 ± 0.004; 0.143 ± 0.004 -> 0.132 ± 0.004 | Helped |
| lower α from 0.1 to 0.001 (ablation) | AudioGenX · AudioCaps validation | Fid_F: 0.398 to 0.138 | Helped |
| lower α from 0.1 to 0.001 (ablation) | AudioGenX · AudioCaps validation | Size: 0.07 to 0.51 | Hurt |
| lower β from 0.1 to 0.001 while α = 0.001 (ablation) | AudioGenX · AudioCaps validation | Fid_F: 0.138 to 0.142 | Hurt |
| lower β from 0.1 to 0.001 while α = 0.001 (ablation) | AudioGenX · AudioCaps validation | Size: 0.426 to 0.517 | Hurt |
| reweight the explanation mask to correct under- or overemphasized text tokens | AudioGenX-guided AudioGen editing · 30 AudioCaps failure cases | FAD: 16.85 -> 2.68 | Helped |
| reweight the explanation mask to correct under- or overemphasized text tokens | AudioGenX-guided AudioGen editing · 30 AudioCaps failure cases | KL_F: 6.45 -> 1.82 | Helped |
Surprising findings
- Attention-based baselines such as AtMan and Grad-CAM-a yield counterfactual results no better than random masks, suggesting they fail to produce necessary explanations for audio generation.
- Negation words ('without', 'without no') receive low importance, which the authors attribute to scarcity of negation in AudioCaps.
Limitations
- Several hyperparameters (alpha, beta) need dataset-specific tuning and trade off fidelity against mask size.
- Explanations may reflect training-data biases.
- Computational time per explanation is higher than some baselines because a mask must be trained per audio token.
Experimental setup
Datasets: AudioCaps
Benchmarks: AudioCaps
Metrics: Fidelity (Fid_F, Fid_CF) KL divergence (KL_F, KL_CF) mask size GPU memory time FAD
Baselines: Random-Mask Grad-CAM-e Grad-CAM-a AtMan Chefer et al.
Builds on: AudioGen AtMan Grad-CAM Chefer et al. transformer explainability Fidelity metric
Closest papers by meaning
- UniAudio: Towards Universal Audio Generation with Large Language Models (ICML 2024 · similarity 0.89)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.88)
- ETTA: Elucidating the Design Space of Text-to-Audio Models (ICML 2025 · similarity 0.87)
- AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation (ICML 2026 · similarity 0.87)
- Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction (ICML 2025 · similarity 0.87)
- UALM: Unified Audio Language Model for Understanding, Generation and Reasoning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.