Research map / Vision-language and multimodal / Human and embodied modeling
AND: Audio Network Dissection for Interpreting Deep Acoustic Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AND, the first LLM-based network dissection framework that describes acoustic neurons in natural language, and uses it for last-layer and middle-layer dissection, audio unlearning via neuron pruning, and analysis of training strategies on neuron polysemanticity.
Problem
Neuron-level interpretability tools exist for vision and language models but not for acoustic models; audio lacks the segmentation/detection resources that vision methods rely on, so natural-language neuron explanations for audio networks are missing.
Approach
An LLM-based pipeline that captions probing audio with an audio LLM (SALMONN), summarizes commonalities of top-K highly and lowly activated clips, calibrates the high-activation summary by removing points that appear in the low-activation summary, and extracts open-set acoustic adjectives via POS tagging and LLM filtering; closed-set concepts are matched via CLIP text-feature similarity to the cap
Key results
- Description-based (DB) closed-concept identification achieves 100% top-1 last-layer dissection accuracy on AST, 76% on BEATs-finetuned, and 58% on BEATs-frozen on ESC-50.
- In human evaluation of 120 middle-layer AST neurons, calibrated summaries (module B) score highest with rating 3.49/5, cosine 0.72, BERTScore 0.46, versus 3.24 for DB and 2.73 for TAB.
- Open-concept pruning yields a target-vs-remaining confidence gap of 3.90 on AST and 1.49 on BEATs-finetuned, and pruning basic adjectives reduces performance more than high-level adjectives by over 10% at 10% pruning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use description-based rather than text-audio-based closed-concept identification | AND Module A on AST · ESC50 | Top-1 Acc: 96.0 -> 100.0 | Helped |
| Use description-based rather than text-audio-based closed-concept identification | AND Module A on BEATs-finetuned · ESC50 | Top-1 Acc: 74.0 -> 76.0 | Helped |
| Use description-based rather than text-audio-based closed-concept identification | AND Module A on BEATs-frozen · ESC50 | Top-1 Acc: 46.0 -> 58.0 | Helped |
| Use calibrated summaries instead of concept identification outputs | AND on AST middle-layer neurons · AST middle-layer neuron human evaluation | Rating: 2.73 -> 3.49 | Helped |
| Use calibrated summaries instead of concept identification outputs | AND on AST middle-layer neurons · AST middle-layer neuron human evaluation | Cos: 0.24 -> 0.72 | Helped |
| Prune neurons identified through open-concept pruning rather than random pruning | AST · ESC50 testing set | ΔA - ΔR: 0.04 -> 3.90 | Helped |
| Prune neurons identified through open-concept pruning rather than random pruning | BEATs-finetuned · ESC50 testing set | ΔA - ΔR: 0.00 -> 1.49 | Helped |
| Prune neurons identified through open-concept pruning rather than random pruning | BEATs-frozen · ESC50 testing set | ΔA - ΔR: -0.01 -> 0.10 | Mixed |
| Prune neurons with the highest number of basic rather than high-level adjectives | AST · ESC50 | testing performance drop | Helped |
| Prune neurons with the most adjectives rather than random neurons | AST · ESC50 | testing performance drop | Mixed |
| Use supervised training rather than SSL pretraining with only final-layer fine-tuning | AST vs BEATs-frozen · ESC50 transformer blocks | percentage of uninterpretable neurons | Mixed |
Surprising findings
- Pruning neurons with the most adjectives does not necessarily reduce performance more than random pruning, contrary to MILAN's visual findings.
- SSL-pretrained BEATs-frozen is much more robust to neuron pruning than supervised models, and its neurons remain uninterpretable across all layers.
Limitations
- Last-layer dissection accuracy declines as the target model's classification accuracy drops (58% for BEATs-frozen).
- Middle-layer evaluation relies on author-written summaries rather than a large-scale annotator study.
- Captioning relies on SALMONN, which lacks batch inference and is slow.
Experimental setup
Datasets: ESC-50 GTZAN
Benchmarks: ESC-50
Metrics: top-1 accuracy top-5 accuracy cosine similarity human rating (1-5) BERTScore confidence change after neuron ablation
Baselines: CLIP-Dissect-style text-audio retrieval (TAB) with CLAP in-context learning concept selection (ICL) MILAN
Builds on: Network Dissection CLIP-Dissect MILAN FALCON
Closest papers by meaning
- Listen, Think, and Understand (ICLR 2024 · similarity 0.86)
- SALMONN: Towards Generic Hearing Abilities for Large Language Models (ICLR 2024 · similarity 0.86)
- The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning (ICML 2025 · similarity 0.86)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.85)
- A Unified Audio Language Model with Text-Aligned Factorized Audio Tokenization (NeurIPS 2026 · similarity 0.85)
- NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.