Research map / Vision-language and multimodal / Cross-modal learning
Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAVS, an audio-visual segmentation framework combining audio-guided grouped modality alignment with contrastive learning and an uncertainty estimation module to handle ambiguous correspondences and sound-state transitions.
Problem
Audio-visual segmentation fails on visually similar objects with different sounding states and on frequent sounding-status changes, leading to over- or under-segmentation.
Approach
Groups visual tokens by density (DPC-KNN), merges group features weighted by audio responsiveness, and applies contrastive learning treating high-response compact features as positives and low-response as negatives; adds a Dirichlet-based uncertainty head that down-weights predictions in high-uncertainty transition regions.
Key results
- Achieves 93.0% J&F on AVS-Objects-S4 and 76.4% on MS3 with HTSAT audio backbone.
- Improves J&F on AVS-Semantic by 4.2% over prior state-of-the-art AVSStone (65.7 vs 61.5).
- On VPO-MSMI, J&F reaches 69.30% vs 58.76% for AVSStone; ablation shows UE adds 1.7 J gain on AVS-Semantic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RAVS instead of CAVP | AVS-S4 segmentation · AVS-Objects-S4 | J &Fm: 90.5 -> 93.5 | Helped |
| Use RAVS instead of BiasAVS | AVS-MS3 segmentation · AVS-Objects-MS3 | J &Fm: 74.0 -> 76.4 | Helped |
| Use RAVS instead of AVSStone | AVS-Semantic segmentation · AVS-Semantic | J &Fm: 61.5 -> 65.7 | Helped |
| Use RAVS instead of CPM* | VPO segmentation · VPO-SS | J &Fβ: 73.49 -> 74.97 | Helped |
| Use RAVS instead of CPM* | VPO segmentation · VPO-MS | J &Fβ: 72.91 -> 73.49 | Helped |
| Use RAVS instead of CPM* | VPO segmentation · VPO-MSMI | J &Fβ: 68.07 -> 69.30 | Helped |
| Add ground-truth mask filtering to the baseline (ablation) | AVS-Objects-MS3 segmentation · AVS-Objects-MS3 | J &Fβ: 64.3 -> 67.9 | Helped |
| Add sound-guided semantic merging to the baseline (ablation) | AVS-Objects-MS3 segmentation · AVS-Objects-MS3 | J &Fβ: 64.3 -> 70.9 | Helped |
| Add contrastive-based alignment to SGSM (ablation) | AVS-Objects-MS3 segmentation · AVS-Objects-MS3 | J &Fβ: 70.9 -> 72.2 | Helped |
| Add uncertainty estimation to AMA (ablation) | AVS-Objects-MS3 segmentation · AVS-Objects-MS3 | J &Fβ: 72.2 -> 76.4 | Helped |
| Add ground-truth mask filtering to the baseline (ablation) | AVS-Semantic segmentation · AVS-Semantic | J &Fβ: 57.2 -> 59.6 | Helped |
| Add sound-guided semantic merging to the baseline (ablation) | AVS-Semantic segmentation · AVS-Semantic | J &Fβ: 57.2 -> 62.6 | Helped |
Limitations
- Gains on AVS-Objects-MS3 and S4 are smaller than on harder subsets; method relies on multi-component design with several hyperparameters (group numbers, thresholds).
- Uncertainty estimation mainly addresses transition frames; limited analysis beyond visualizations.
Experimental setup
Datasets: AVS-Objects AVS-Semantic VPO
Benchmarks: AVS-Objects-S4 AVS-Objects-MS3 AVS-Semantic VPO-SS VPO-MS VPO-MSMI
Metrics: J&F J (mean IoU) F-score (beta=0.3)
Baselines: TPAVI AVSC CATR DiffusionAVS ECMVAE AuTR AQFormer AVSegFormer AVSBG GAVS BAVS COMBO
Builds on: CAVP AVSStone AVSegFormer CATR
Closest papers by meaning
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.92)
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.92)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.91)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.91)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.90)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.