Research map / Detection and segmentation / CNN features
DeepASA: An Object-Oriented Multi-Purpose Network for Auditory Scene Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepASA, a unified object-oriented network that jointly performs MIMO source separation, dereverberation, SED, classification and DoA estimation, refined by a chain-of-inference module with temporal coherence matching.
Problem
Auditory scene analysis requires separating overlapping, moving sound sources and estimating their class, activity and direction, but task-specific models lack cross-task relational reasoning and track-wise processing causes parameter association ambiguity.
Approach
Encapsulate features into per-object representations (object-oriented processing) so all task decoders share object-wise permutation, then refine objects iteratively via a chain-of-inference that cross-attends SED and DoA cues (temporal coherence matching) and fuses them with FiLM into the object features; also uses a time-varying learnable Gaussian window STFT front end.
Key results
- Chain-of-inference on ASA2 improves USS to 11.2 dB SI-SDRi and SELD score from 0.230 to 0.206 over the no-CoI model.
- On MC-FUSS after fine-tuning from ASA2, DeepASA (SEP) reaches 18.5 dB total SI-SDRi versus 16.6 for DeFT-Mamba.
- On STARSS23, DeepASA with CoI reaches SELD score 0.253, better than the 1st-ranked NERC-SLIP system with ensemble (0.260).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-inference (ablation) | DeepASA · ASA2 | SI-SDRi: 11.0 -> 11.2 | Helped |
| add chain-of-inference (ablation) | DeepASA · ASA2 | SELD: 0.230 -> 0.206 | Helped |
| use only the SED query branch and remove audio decoder 2 (ablation) | DeepASA chain-of-inference · ASA2 | SELD: 0.206 -> 0.221 | Hurt |
| use only the DoA query branch and remove audio decoder 2 (ablation) | DeepASA chain-of-inference · ASA2 | SELD: 0.206 -> 0.228 | Hurt |
| use both SED and DoA query branches but remove audio decoder 2 (ablation) | DeepASA chain-of-inference · ASA2 | SELD: 0.206 -> 0.214 | Hurt |
| remove pre-trained ATST (ablation) | DeepASA (SEP) · MC-FUSS total | SI-SDRi: 18.5 -> 18.4 | Hurt |
| remove pre-trained ATST (ablation) | DeepASA · ASA2 | SELD: 0.230 -> 0.240 | Hurt |
| replace the time-varying learnable window with a time-invariant learnable window (ablation) | DeepASA · ASA2 | SELD: 0.230 -> 0.238 | Hurt |
| replace STFT with learnable 1D convolution kernels (ablation) | DeepASA · ASA2 | SELD: 0.237 -> 0.338 | Hurt |
| add ATST and T-CRNN to the SED decoder (ablation) | DeepASA · ASA2 | SELD: 0.301 -> 0.276 | Helped |
| add F-CRNN to the ATST + T-CRNN SED decoder (ablation) | DeepASA · ASA2 | SELD: 0.276 -> 0.266 | Helped |
| add a noise decoder (ablation) | DeepASA · ASA2 | SI-SDRi: 10.3 -> 11.0 | Helped |
Surprising findings
- Adding the noise decoder improved classification recall for domestic sounds from 49.4% to 64.9%, suggesting explicit noise estimation helps classification.
- Repeating the chain-of-inference a second stage degraded SELD (0.206 to 0.216), so a single CoI pass performed best.
Limitations
- Large parameter count from the pre-trained ATST in the SED decoder.
- Evaluated only on simulated reverberation times of 0.2-0.6 s and SNR of 6-30 dB; degrades for longer reverberation or negative SNR.
- Performance drops sharply when there are more sources than the maximum (6-7 sources vs. 5 max).
Experimental setup
Datasets: ASA2 MC-FUSS STARSS23 AudioSet TAU-SNoise DB
Benchmarks: ASA2 MC-FUSS STARSS23
Metrics: SI-SDRi SDRi ER F1 LE LR SELD score parameter count MAC/s
Baselines: DeFT-Mamba ByteDance-uss MC-BSRNN TF-GridNet DeFTAN-II SpatialNet SELDNet EINV2 ResNet Conformer SELD-Mamba MFF-EINV2 CST-former
Builds on: DeFT-Mamba ATST SELDNet FiLM TDY-CNN
Closest papers by meaning
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.87)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.86)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.86)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.86)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.86)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.