Research map / Label-efficient and robust learning / Recognition and long-tail
Listenable Maps for Audio Classifiers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L-MAC, a posthoc interpreter for audio classifiers that trains a decoder on classifier latents to output a binary mask over the linear STFT spectrogram, yielding faithful and listenable audio explanations.
Problem
Posthoc interpretation of audio classifiers that operate on mel-spectrogram-like features is hard because saliency maps on those features are not easily understandable and cannot be inverted to audio.
Approach
A decoder attached to a frozen pretrained classifier's intermediate representations predicts a binary mask applied to the linear STFT magnitude (with original phase), trained with a loss that keeps the masked-in audio confidently classified and the masked-out audio not, so the output can be inverted to a listenable waveform while staying feature-agnostic.
Key results
- In-domain ESC-50 (Mel), L-MAC reaches AI 35.63 and AD 1.59 versus best gradient baseline GradCAM AI 7.00 and AD 10.93.
- Out-of-domain ESC-50 (Mel), L-MAC reaches AI 60.25, AD 4.84, Fid-In 0.80 versus GradCAM AI 19.88, AD 18.85.
- In a 15-participant user study, L-MAC MOS-1 59.13 and MOS-2 64.00 exceed L2I's 55.1 and 50.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use L-MAC instead of gradient-based and L2I explanations | CNN14 audio classifier · ESC50 in-domain, STFT-Mel | AI: 8.50 -> 36.25 | Helped |
| use L-MAC instead of the strongest baseline | CNN14 audio classifier · ESC50 out-of-domain, STFT-Mel | FF: 0.25 -> 0.39 | Helped |
| use L-MAC instead of the strongest baseline | CNN14 audio classifier · ESC50 in-domain, STFT-Mel | FF: 0.11 -> 0.20 | Helped |
| use L-MAC instead of the strongest baseline | audio classifier · ESC50 white-noise contamination | AI: 8.87 -> 83.62 | Helped |
| use L-MAC instead of L2I | audio interpretations · L2I showcase recordings, MOS-1 | MOS: 55.1 -> 59.13 | Helped |
| add fine-tuning with lambda_g=4 (ablation) | L-MAC · ESC50 in-domain, STFT-Mel | AI: 36.25 -> 32.37 | Hurt |
| add fine-tuning with lambda_g=16 (ablation) | L-MAC · ESC50 in-domain, STFT-Mel | AI: 36.25 -> 27.12 | Hurt |
| add fine-tuning with lambda_g=32 (ablation) | L-MAC · ESC50 in-domain, STFT-Mel | AI: 36.25 -> 23.00 | Hurt |
| add fine-tuning with lambda_g=4 (ablation) | L-MAC · ESC50 out-of-domain, STFT-Mel | AI: 60.63 -> 50.75 | Hurt |
| add fine-tuning with lambda_g=16 (ablation) | L-MAC · ESC50 out-of-domain, STFT-Mel | AI: 60.63 -> 37.62 | Hurt |
| add fine-tuning with lambda_g=32 (ablation) | L-MAC · ESC50 out-of-domain, STFT-Mel | AI: 60.63 -> 28.88 | Hurt |
| increase CCT to 0.7 at lambda_g=32 (ablation) | L-MAC fine-tuning · ESC50 out-of-domain, STFT-Mel | AI: 28.88 -> 38.50 | Helped |
Surprising findings
- Fine-tuning for audio quality (lambda_g) trades off faithfulness; the most faithful results come without fine-tuning.
- GradCAM remains insensitive to classifier weight randomization in the cascading test, raising sanity-check concerns for it.
Limitations
- Fine-tuning for listenability reduces faithfulness scores.
- Evaluation limited to ESC-50 and UrbanSound8k with a single CNN14 classifier architecture.
- User study involves only 15 participants and a small set of recordings.
Experimental setup
Datasets: ESC-50 WHAM! noise VGG-Sound UrbanSound8k LJSpeech
Benchmarks: ESC-50 UrbanSound8k
Metrics: Average Increase Average Drop Average Gain Faithfulness on Spectra Input Fidelity Sparseness Complexity Mean Opinion Score ROAR Model Randomization Test SSIM
Baselines: Saliency SmoothGrad Integrated Gradients GradCAM Guided GradCAM Guided Backprop SHAP L2I
Builds on: Listen-to-Interpret (L2I) Extremal perturbations / masking-based interpretation SimCLR
Closest papers by meaning
- Listenable Maps for Zero-Shot Audio Classifiers (NeurIPS 2024 · similarity 0.91)
- Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data (ICLR 2025 · similarity 0.85)
- AND: Audio Network Dissection for Interpreting Deep Acoustic Models (ICML 2024 · similarity 0.83)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.83)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.83)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.