Research map / Vision-language and multimodal / Cross-modal learning
Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds cost-effective audio-visual segmentation benchmarks (VPO) by pairing COCO images with VGGSound audio, and proposes CAVP, a supervised contrastive method that mines informative audio-visual pairs via randomly permuted audio-visual pairing.
Problem
Audio-visual segmentation (AVS) suffers from biased datasets (commonsense sound-source bias, few multi-instance cases) and models that generalise poorly, plus instance-based contrastive learning that fails to mine informative negatives for cross-modal alignment.
Approach
Create new AVS benchmarks by matching COCO semantic-segmentation images with VGGSound audio by class (with synthesized stereo spatial cues), and train with a supervised audio-visual contrastive loss over randomly re-paired cross-attention features so that many positive and negative audio-visual combinations are mined within a batch.
Key results
- On resized AVSBench-Object (SS) with ResNet50, CAVP reaches 85.77 mIoU, +7.81 over the second-best method.
- On original-resolution AVSBench-Semantics, CAVP reaches 50.75 mIoU vs 43.39 for TPAVI and 41.48 for AVSegFormer.
- On VPO-MSMI, CAVP reaches 60.36 mIoU, improving over the best baseline by at least 6.14 mIoU points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAVP instead of the strongest listed baseline | AVS segmentation with ResNet50 · AVSBench-Object (SS), 224×224 | mIoU: 77.96 -> 85.77 | Helped |
| Use CAVP instead of the strongest listed baseline | AVS segmentation with ResNet50 · AVSBench-Object (MS), 224×224 | mIoU: 55.70 -> 62.39 | Helped |
| Use CAVP instead of the strongest listed baseline | AVS segmentation with ResNet50 · AVSBench-Semantics, 224×224 | mIoU: 24.93 -> 44.70 | Helped |
| Use CAVP instead of the strongest listed baseline | AVS segmentation with ResNet50 · AVSBench-Semantics, original resolution | mIoU: 43.39 -> 50.75 | Helped |
| Replace TPAVI cross-attention with fCA (ablation) | CAVP · AVSBench-Semantics and VPO (MSMI) | mIoU: 42.74 -> 47.18; 52.83 -> 58.50 | Helped |
| Add supervised contrastive learning without informative-sample selection (ablation) | CAVP cross-attention model · AVSBench-Semantics and VPO (MSMI) | mIoU: 47.18 -> 48.72; 58.50 -> 59.35 | Helped |
| Add CAVP informative-sample selection (ablation) | Supervised-contrastive AVS model · AVSBench-Semantics and VPO (MSMI) | mIoU: 48.72 -> 50.75; 59.35 -> 64.31 | Helped |
| Use sigmoid rather than min-max normalization in fCA (ablation) | CAVP · AVSBench-Semantics | mIoU: +2.38% | Helped |
| Balance positive and negative samples at 50% each (ablation) | CAVP contrastive sampling · AVSBench-Semantics | mIoU: 1.87%; 1.56% | Helped |
| Replace original AVSBench-Object audio with VGGSound audio from the same class (ablation) | TPAVI · AVSBench-Object (SS) versus AVS-M-SS | mIoU: 72.01 ± 0.7% -> 72.07 ± 0.7% | No clear effect |
Surprising findings
- Replacing AVSBench audio with same-class VGGSound audio yields equivalent TPAVI performance (p-values 0.66 mIoU and 0.73 F-beta), suggesting the VPO construction is valid.
- Softmax-based cross-attention activation performed worst among the tested options, while sigmoid was best.
Limitations
- VPO dataset lacks temporal information.
- VPO may exhibit class imbalance similar to AVSBench-Semantics.
- Spatial audio simulation does not model visual depth or arrival time differences.
Experimental setup
Datasets: AVSBench-Object AVSBench-Semantics COCO VGGSound VPO-SS VPO-MS VPO-MSMI
Benchmarks: AVSBench-Object AVSBench-Semantics VPO-SS VPO-MS VPO-MSMI
Metrics: mIoU F-beta FDR
Baselines: TPAVI AVSegFormer CATR AuTR AQFormer AVSC BAVS AVSBG ECMVAE DiffusionAVS
Builds on: TPAVI AVSegFormer supervised contrastive learning (SupCon) MaskFormer DeepLabV3+
Closest papers by meaning
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.92)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.92)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.92)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.91)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.91)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.