Research map / Vision-language and multimodal / Cross-modal learning
Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDESeg, an audio-visual segmentation framework that derives single-source audio semantics from mixed audio, enhances their discriminability with intra-class refinement, and eliminates visually unmatched audio tokens via visual-guided scoring.
Problem
Audio-visual segmentation is hampered by feature confusion from overlapping sound sources in mixed audio and by audio-visual matching difficulty caused by large intra-class variation of sounds from the same object.
Approach
Instead of separating mixed audio, derive multiple source-specific audio representations by retrieving nearest semantic centers from a single-source audio memory and computing edge-based compensations; refine them with intra-class discriminative features via scaling; and filter out non-matching audio tokens using a Gumbel-softmax visual clustering and relevance scoring module.
Key results
- Achieves J&F of 67.9% on AVS-Semantic vs. 61.5% for the second-best method (AVSStone), and 94.2% on AVS-Objects-S4 vs. 90.5% (CAVP).
- On VPO, DDESeg with HTSAT audio reaches 74.38 J&F on VPO-SS, 74.30 on VPO-MS, and 68.39 on VPO-MSMI, beating AVSStone by 6%, 10.04%, and 4.9% on the three subsets.
- Ablation on AVS-Semantic: baseline 58.00 J&F rises to 67.85 with derivation, discriminative enhancement, and elimination modules.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDESeg with Trans. and HTSAT instead of the strongest reported baseline | DDESeg · AVS-Objects-S4 | J &Fm: 90.5% to 94.2% | Helped |
| Use DDESeg with Trans. and HTSAT instead of the strongest reported baseline | DDESeg · AVS-Objects-MS3 | J &Fm: 74.0% to 77.9% | Helped |
| Use DDESeg with Trans. and HTSAT instead of the strongest reported baseline | DDESeg · AVS-Semantic | J &Fm: 61.5% to 67.9% | Helped |
| Use DDESeg with Trans. and HTSAT instead of the strongest reported baseline | DDESeg · VPO-SS | J &Fβ: 68.54 to 74.38 | Helped |
| Use DDESeg with Trans. and HTSAT instead of the strongest reported baseline | DDESeg · VPO-MS | J &Fβ: 64.26 to 74.30 | Helped |
| Use DDESeg with Trans. and HTSAT instead of the strongest reported baseline | DDESeg · VPO-MSMI | J &Fβ: 58.76 to 68.39 | Helped |
| Add audio representation derivation (ablation) | DDESeg baseline · AVS-Objects-MS3 | J &Fβ: 65.81 -> 71.51 | Helped |
| Add audio representation derivation (ablation) | DDESeg baseline · AVS-Semantic | J &Fβ: 58.00 -> 62.60 | Helped |
| Add discriminative semantic enhancement (ablation) | DDESeg with derivation · AVS-Objects-MS3 | J &Fβ: 71.51 -> 73.40 | Helped |
| Add discriminative semantic enhancement (ablation) | DDESeg with derivation · AVS-Semantic | J &Fβ: 62.60 -> 65.55 | Helped |
| Add dynamic elimination (ablation) | DDESeg with derivation and discriminative enhancement · AVS-Objects-MS3 | J &Fβ: 73.40 -> 75.57 | Helped |
| Add dynamic elimination (ablation) | DDESeg with derivation and discriminative enhancement · AVS-Semantic | J &Fβ: 65.55 -> 67.85 | Helped |
Surprising findings
- Replacing VGGish with HTSAT as audio backbone gives only modest gains of 1.5% J and 0.7% F-score on AVS-Objects.
- A plain FC+Sigmoid filtering scheme degrades performance relative to no elimination, showing naive filtering hurts.
Limitations
- Relies on a pre-trained audio classification model (HTSAT) and a memory of single-source audio features, which requires single-source audio for construction.
- Performance on VPO is reported at 224x224 resolution, and CPM comparison uses original resolution, complicating comparison.
Experimental setup
Datasets: AVS-Objects AVS-Semantic VPO AudioSet
Benchmarks: AVS-Objects-S4 AVS-Objects-MS3 AVS-Semantic VPO-SS VPO-MS VPO-MSMI
Metrics: J&F J (Jaccard) F-score
Baselines: TPAVI AVSC CATR AVSegFormer COMBO CAVP AVSStone BiasAVS QDFormer CPM
Builds on: Mix Transformer (MiT) HTSAT Generalized Laplace Operator CAVP AVSegFormer
Closest papers by meaning
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.91)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.91)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.91)
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.90)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.89)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.