Research map / Detection and segmentation / Open-set and few-shot detection
QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QDFormer, which decomposes multi-source audio semantics via product quantization with a shared codebook and uses global-to-local quantization distillation to improve audiovisual segmentation in noisy, multi-source videos.
Problem
Audiovisual segmentation (AVS) struggles when multiple sound sources entangle and background noise disturbs audio, and when sound events change frequently across frames, making audio-visual correspondence unreliable.
Approach
Assume independent sound events so the multi-source audio semantic space is a Cartesian product of single-source spaces; decompose audio into several tokens quantized with a shared codebook (product quantization as an information bottleneck), and distill stable clip-level quantized semantics into frame-level tokens via a frozen shared codebook.
Key results
- On AVS-Semantic with ResNet-50, QDFormer reaches 46.6 mIoU, exceeding AOT (25.4) by 21.2 mIoU.
- On AVS-Object-Multi with ResNet-50, QDFormer achieves 61.6 J&F, outperforming AVSegFormer (56.2) by 5.4 and CATR (59.4) by 2.2.
- Ablation: removing decomposition drops AVS-Semantic mIoU from 46.6 to 33.5; quantization-based decomposition beats non-quantized variant by 2.7 mIoU (42.5 vs 39.4 before recombination and calibration).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add quantization-based semantic decomposition (ablation) | QDFormer · AVS-Object-Multi | J &F: 57.6 -> 59.7 | Helped |
| add quantization-based semantic decomposition (ablation) | QDFormer · AVS-Semantic | mIoU: 39.4 -> 42.5 | Helped |
| add audiovisual semantic recombination (ablation) | QDFormer with quantization-based semantic decomposition · AVS-Object-Multi | J &F: 59.7 -> 60.1 | Helped |
| add audiovisual semantic recombination (ablation) | QDFormer with quantization-based semantic decomposition · AVS-Semantic | mIoU: 42.5 -> 44.5 | Helped |
| add local calibration (ablation) | QDFormer with quantization-based semantic decomposition and audiovisual semantic recombination · AVS-Object-Multi | J &F: 60.1 -> 61.6 | Helped |
| add local calibration (ablation) | QDFormer with quantization-based semantic decomposition and audiovisual semantic recombination · AVS-Semantic | mIoU: 44.5 -> 46.6 | Helped |
| decompose audio semantics rather than interact directly with mixed audio (ablation) | QDFormer · AVS-Object-Multi | J &F: 52.9 -> 61.6 | Helped |
| decompose audio semantics rather than interact directly with mixed audio (ablation) | QDFormer · AVS-Semantic | mIoU: 33.5 -> 46.6 | Helped |
| use semantic-domain decomposition instead of off-the-shelf time-domain audio separation (ablation) | QDFormer · AVS-Object-Multi | J &F: 56.2 -> 61.6 | Helped |
| use semantic-domain decomposition instead of off-the-shelf time-domain audio separation (ablation) | QDFormer · AVS-Semantic | mIoU: 38.9 -> 46.6 | Helped |
| use quantized and decomposed audio representation rather than conventional continuous representation | QDFormer · AVS-Semantic with background noise | mIoU | Helped |
Surprising findings
- Decomposing audio in the semantic domain outperforms decomposing it in the time domain with an off-the-shelf sound separation model (Sep. model gains 5.5 mIoU vs 13.1 for full method over baseline).
Limitations
- Multi-source gains are much larger than single-source gains, suggesting limited benefit for simple single-source audio.
- Relies on assumption of independent sound events for the Cartesian product decomposition.
- Codebook size set to roughly the number of sound categories, requiring prior knowledge of label set size.
Experimental setup
Datasets: AVS-Object AVS-Semantic AudioSet (noise for robustness analysis)
Benchmarks: AVS-Object-Single AVS-Object-Multi AVS-Semantic
Metrics: J&F J F mIoU
Baselines: AVS AVSegFormer CATR AOT 3DC LVS MSSL Bi-Gen iGAN SST LGVT
Builds on: AVS (Zhou et al.) AVSegFormer product quantization VQ-VAE Mask2Former-style transformer decoder
Closest papers by meaning
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.89)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.88)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.87)
- Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation (AAAI 2025 · similarity 0.87)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.87)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.