Research map / Detection and segmentation / Open-set and few-shot detection
Unsupervised Audio-Visual Segmentation with Modality Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoCA, an unsupervised audio-visual segmentation method that adapts frozen ImageBind with a lightweight audio-visual adapter trained via contrastive pixel matching over DINO-mined image pairs, refined with SAM masks.
Problem
Audio-visual segmentation (AVS) currently needs costly pixel-level mask-audio annotations; the paper targets pixel-level sounding-object segmentation without any audio-mask pairs.
Approach
Mine positive and negative image pairs with DINO KNN, train only a small audio-visual adapter (AdaAV) on frozen ImageBind with a pixel matching aggregation loss (SSD+NCC) inside an image-level contrastive objective, then refine clusters with SAM/OWOD mask proposals.
Key results
- MoCA reaches 0.68 mIoU on S4 and 0.57 on MS3 with no audio-mask pairs, versus 0.58 and 0.34 for the best training-free baseline OWOD-BIND.
- On AVSS, MoCA achieves 0.31 mIoU versus 0.26 for OWOD-BIND.
- The supervised AVSBench reaches 0.54 mIoU on MS3, which is below MoCA's 0.57 on that split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MoCA instead of OWOD-BIND | AVSBench S4 · AVSBench S4 test | mIoU: 0.58 -> 0.68 | Helped |
| use MoCA instead of OWOD-BIND | AVSBench MS3 · AVSBench MS3 test | mIoU: 0.34 -> 0.57 | Helped |
| use MoCA instead of OWOD-BIND | AVSS · AVSS test | mIoU: 0.26 -> 0.31 | Helped |
| add pixel matching aggregation (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.34 -> 0.57 | Helped |
| add mask proposal matching (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.52 -> 0.57 | Helped |
| combine SSD and NCC instead of using SSD only (ablation) | MoCA · AVSBench S4 test | mIoU: 0.61 -> 0.68 | Helped |
| combine SSD and NCC instead of using NCC only (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.44 -> 0.57 | Helped |
| increase AdaAV blocks from 2 to 6 (ablation) | MoCA · AVSBench S4 test | mIoU: 0.60 -> 0.68 | Helped |
| increase AdaAV blocks from 2 to 6 (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.39 -> 0.57 | Helped |
| place AdaAV blocks in the last-k image blocks rather than interleaved (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.56 -> 0.57 | No clear effect |
| set triplet-loss margin to 0.3 rather than 0.1 (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.54 -> 0.57 | Helped |
| set triplet-loss margin to 0.3 rather than 0.5 (ablation) | MoCA · AVSBench MS3 test | mIoU: 0.56 -> 0.57 | Helped |
Surprising findings
- MoCA's MS3 mIoU (0.57) exceeds the supervised AVSBench baseline (0.54) reported in the same table, although the text says it falls short by 0.20 on MS3.
- Training with SSD alone or NCC alone gives much lower results than SSD+NCC, with SSD alone at 0.40 MS3 mIoU.
Limitations
- Relies on frozen foundation models (ImageBind, DINO, SAM, OWOD), so performance depends on their quality.
- Mask proposal matching uses an empirically tuned IoU threshold (0.5) and k-means clustering.
Experimental setup
Datasets: VGGSound AVSBench AVSS
Benchmarks: AVSBench S4 AVSBench MS3 AVSS VGGSound-Single
Metrics: mIoU F-score IoU@0.5 AUC
Baselines: AT-GDINO-SAM SAM-BIND OWOD-BIND AVSBench AV-SAM SAMA-AVS DG-SCT LVS MSSL EZ-VSL Mix-Localize iGAN
Builds on: ImageBind DINO SAM Segment Anything AVSBench
Closest papers by meaning
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.92)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.92)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.90)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.90)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.89)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.