Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Listenable Maps for Zero-Shot Audio Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LMAC-ZS, a decoder-based post-hoc explainer that produces listenable saliency masks for zero-shot audio classifiers (CLAP) using a similarity-preserving loss, evaluated with masking faithfulness metrics on ESC50 and UrbanSound8K.
Problem
Explaining the decisions of zero-shot audio classifiers, whose class labels are free-form text prompts and whose decisions are based on audio-text similarity, has not been addressed by post-hoc explanation methods.
Approach
A text-conditioned decoder (cross-attention over CLAP CNN14 representations) is trained with a loss that keeps the audio-text similarity after masking the input, plus a sparsity term and a diversity term that ties mask differences to text-prompt differences, so the explanation depends on the prompt.
Key results
- On ESC50 with Mel-masking, LMAC-ZS (Full) reaches AI 23.45 and AD 17.12 versus GradCAM++ AI 8.45 and AD 35.07 (In-Domain).
- With STFT masking on ESC50, LMAC-ZS (Full) achieves AI 43.35 and AD 4.29 versus the best baseline GradCAM++ AI 32.50 and AD 8.97.
- FF is comparable across methods, and GradCAM++ obtains better FF in several out-of-domain settings, with larger masks (higher Mask-Mean).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LMAC-ZS instead of GradCam++ for explanation generation | CLAP zero-shot classifier · ESC50 and UrbanSound8K, clean audio, Mel- and STFT-Masking | AI: ESC50 Mel: 8.45 -> 29.00 (CT); ESC50 STFT: 32.50 -> 43.35 (Full); UrbanSound8K Mel: 7.21 -> 35.69 (Full); UrbanSound8K S | Helped |
| Use LMAC-ZS instead of GradCam++ for explanation generation | CLAP zero-shot classifier · ESC50 with background contamination | AI: ESC50 contamination Mel: 9.82 -> 20.43; white noise Mel: 7.12 -> 28.52 (CT); LJ-Speech Mel: 12.85 -> 24.38 (CT) | Mixed |
| Train LMAC-ZS on all CLAP datasets rather than Clotho only | LMAC-ZS · ESC50, clean audio, STFT-Masking | AI: 37.40 -> 43.35 | Helped |
| Train LMAC-ZS on all CLAP datasets rather than Clotho only | LMAC-ZS · ESC50, white-noise contamination, Mel-Masking | AI: 28.52 -> 14.25 | Hurt |
| Train LMAC-ZS on all CLAP datasets rather than Clotho only | LMAC-ZS · ESC50, LJ-Speech contamination, Mel-Masking | AI: 24.38 -> 8.95 | Hurt |
Surprising findings
- GradCAM++ obtains better FF numbers than LMAC-ZS in several settings, though its masks are larger.
- Training only on Clotho gives comparable or better results than training on the full CLAP data in some settings.
Limitations
- Implementation restricted to fixed-length audio.
- Evaluation limited to the CLAP model; few alternatives exist.
- Faithfulness metrics analyze only the dominant class contribution.
Experimental setup
Datasets: Clotho FSD50K AudioCaps MACS ESC50 UrbanSound8K LJ-Speech
Benchmarks: ESC50 UrbanSound8K
Metrics: Average Increase Average Drop Average Gain Faithfulness on Spectra Input Fidelity Sparseness Complexity Mask-Mean
Baselines: GradCAM GradCAM++ SmoothGrad Integrated Gradients
Builds on: L-MAC CLAP Stable Diffusion cross-attention
Closest papers by meaning
- Listenable Maps for Audio Classifiers (ICML 2024 · similarity 0.91)
- Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.85)
- ZeroSep: Separate Anything in Audio with Zero Training (NeurIPS 2025 · similarity 0.84)
- AND: Audio Network Dissection for Interpreting Deep Acoustic Models (ICML 2024 · similarity 0.84)
- AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound (NeurIPS 2025 · similarity 0.83)
- CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.