Research map / Video / Video segmentation and tracking
Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COMBO, an audio-visual transformer for audio-visual segmentation that models pixel, modality, and temporal "bilateral entanglements" via a Siam-Encoder Module, Bilateral-Fusion Module, and adaptive inter-frame consistency loss.
Problem
Audio-visual segmentation (AVS) requires grouping pixels of sounding objects in video, which demands aligning audio with pixel-level visual features, modeling temporal consistency across frames, and handling noisy image features.
Approach
Treats AVS through three bilateral entanglements: a Siam-Encoder that fuses SAM-style class-agnostic "Maskige" prior maps into visual features, a bidirectional audio-visual attention fusion (audio-to-visual and visual-to-audio), and an adaptive inter-frame consistency loss weighted by the similarity of adjacent-frame predicted masks.
Key results
- COMBO with PVT-v2 reaches 84.7 mIoU on S4 and 59.2 mIoU on MS3 on AVSBench-object, surpassing prior SOTA by 2.6 and 0.2 mIoU.
- On AVSBench-semantic (AVSS), COMBO-PVT achieves 42.1 mIoU, up 5.4 over AVSegFormer-PVT (36.7).
- Ablations show removing the Siam-Encoder costs 1.1 mIoU on S4 and 2.2 on AVSS (MJ), and the inter-frame loss contributes 0.7 mIoU on S4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use COMBO with PVT-v2 instead of AVSegFormer with PVT-v2 | AVSBench-object S4 · AVSBench-object S4 | MJ: +2.6 | Helped |
| use COMBO with PVT-v2 instead of AVSegFormer with PVT-v2 | AVSBench-object S4 · AVSBench-object S4 | MF: +2.0 | Helped |
| use COMBO with PVT-v2 instead of CATR with PVT-v2 | AVSBench-object MS3 · AVSBench-object MS3 | MJ: +0.2 | Helped |
| use COMBO with PVT-v2 instead of DiffusionAVS with PVT-v2 | AVSBench-object MS3 · AVSBench-object MS3 | MF: +1.2 | Helped |
| use COMBO with PVT-v2 instead of AVSegFormer with PVT-v2 | AVSBench-semantic AVSS · AVSBench-semantic AVSS | MJ: +5.4 | Helped |
| use COMBO with PVT-v2 instead of AVSegFormer with PVT-v2 | AVSBench-semantic AVSS · AVSBench-semantic AVSS | MF: +4.1 | Helped |
| add the Siam-Encoder Module (ablation) | COMBO · AVSS | MJ: 31.9 -> 33.3 | Helped |
| add the Bilateral-Fusion Module (ablation) | COMBO · S4 | MJ: 81.1 -> 81.7 | Helped |
| add adaptive inter-frame consistency loss (ablation) | COMBO · S4 | MJ: 81.0 -> 81.7 | Helped |
| use separate rather than shared Siam-Encoder weights (ablation) | COMBO SEM · AVSS | MJ: 31.1 -> 33.3 | Helped |
| use bilateral fusion rather than no fusion (ablation) | COMBO · AVSS | MJ: 33.1 -> 33.3 | Helped |
| use bilateral fusion rather than visual fusion only (ablation) | COMBO · AVSS | MJ: 32.6 -> 33.3 | Helped |
Surprising findings
- Sharing weights between the image and Maskige encoders in SEM performs nearly as well as separate weights (81.5 vs 81.7 mIoU on S4) with far fewer parameters.
Limitations
- Excessively high inter-frame consistency weight degrades performance, since videos are not fully consistent.
- Relies on an external frozen proposal generator (Semantic-SAM) to produce class-agnostic masks.
Experimental setup
Datasets: AVSBench-object AVSBench-semantic
Benchmarks: AVSBench-object S4 AVSBench-object MS3 AVSBench-semantic AVSS
Metrics: mIoU (Jaccard J) F-score
Baselines: AVSBench AVSegFormer BAVS CATR AVS-BiGen ECMVAE DiffusionAVS CSMF LVS MSSL
Builds on: Mask2Former MaskFormer AVSBench Semantic-SAM VGGish
Closest papers by meaning
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.88)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.88)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.87)
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.87)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.87)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.