Research map / Vision-language and multimodal / Cross-modal learning
Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GAVS, an encoder-prompt-decoder audio-visual segmentation model that prompts SAM with semantic-aware audio prompts and tunes it with a correlation adapter, aiming at zero-shot, few-shot and cross-dataset generalization.
Problem
Audio-visual localization and segmentation models based on encoder-fusion-decoder generalize poorly to unseen classes and to different datasets when training data is scarce.
Approach
Instead of fusing audio and visual features and decoding masks, audio is used as a prompt for a frozen-ish visual foundation model (SAM). A Semantic-aware Audio Prompt (visual cues + learnable adaptive noise + audio, aligned by a triplet loss) drives the mask decoder, and a Correlation Adapter (ColA) tunes the prompt-context features in cross-modal attention so that prior knowledge of SAM is prese
Key results
- On AVS-V3 0-shot, GAVS reaches 54.71 mIoU versus 54.26 for AVSegFormer; with 3-shot it surpasses AVSBench/AVSegFormer trained with 5-shot.
- On VGG-SS (cross-dataset, zero-shot), GAVS achieves 41.07 cIoU, above AVSegFormer (38.86) and AVSBench (36.86), though below the best in-set model AVIN-RN (44.90).
- On AVS-Benchmarks V2, GAVS reaches 67.70 mIoU and 0.788 F-score, the best among compared methods, but it underperforms on V1S (80.06 vs 82.06 mIoU for AVSegFormer).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the encoder-prompt-decoder method instead of AV fusion (ablation) | GAVS · AVS-V2 | mIoU (%): 59.69 -> 63.35 | Helped |
| Add ColA instead of tuning AV and VA cross-modal attention with adapters (ablation) | GAVS audio source decoder · AVS-V2 | mIoU (%): 62.09 -> 63.35 | Helped |
| Add adapters to the visual backbone (ablation) | GAVS audio-prompt model · AVS-V2 | mIoU (%): 63.35 -> 65.92 | Helped |
| Add SAP (ablation) | GAVS with visual backbone adapters · AVS-V2 | mIoU (%): 65.92 -> 67.70 | Helped |
| Fine-tune the decoder instead of freezing it (ablation) | GAVS audio source decoder · AVS-V2 | mIoU (%): 57.41 -> 62.08 | Helped |
| Tune AV cross-modal attention with adapters instead of freezing the decoder (ablation) | GAVS audio source decoder · AVS-V2 | mIoU (%): 57.41 -> 62.11 | Helped |
| Tune VA cross-modal attention with adapters instead of freezing the decoder (ablation) | GAVS audio source decoder · AVS-V2 | mIoU (%): 57.41 -> 61.84 | Helped |
| Use ColA instead of freezing the decoder (ablation) | GAVS audio source decoder · AVS-V2 | mIoU (%): 57.41 -> 63.35 | Helped |
| Add AV- and VA-attention adapters alongside ColA (ablation) | GAVS audio source decoder · AVS-V2 | mIoU (%): 63.35 -> 62.09 | Hurt |
| Use GAVS rather than the strongest zero-shot AVS baseline | AVS-V3 0-shot · AVS-V3 0-shot | mIoU (%): 54.26 -> 54.71 | Helped |
| Use GAVS rather than the strongest fusion baseline | VGG-SS-Sub 5-shot · VGG-SS-Sub 5-shot | cIoU (%): 57.65 -> 60.14 | Helped |
| Use GAVS rather than the strongest in-set baseline | VGG-SS zero-shot · VGG-SS zero-shot | cIoU (%): 44.90 -> 41.07 | Hurt |
Surprising findings
- On AVS-Benchmarks V1S, GAVS is below AVSegFormer (80.06 vs 82.06 mIoU), so the audio-prompt paradigm does not win everywhere in the in-domain setting.
- Audio-SAM with prompts only reaches 56.33 mIoU on V1S, showing that the audio prompt alone is insufficient without the visual adapters and SAP.
Limitations
- Performance on the single-source V1S benchmark is lower than the best fusion-based method.
- Zero-shot VGG-SS performance still lags behind the best in-set trained AVL model (AVIN-RN, 44.90 cIoU).
- Experiments are on a limited set of datasets, with AVS-V3 being a self-constructed split.
Experimental setup
Datasets: AVS-Benchmarks (V1S, V1M, V2) AVS-V3 VGG-SS VGG-SS-Sub VGG-Sound
Benchmarks: AVS-Benchmarks AVS-V3 VGG-SS VGG-SS-Sub
Metrics: mIoU F-score cIoU AUC
Baselines: AVSBench AVSegFormer AVSC AuTR AV-SAM SAM-Fusion Audio-SAM SLAVC EZ-VSL AVIN-RN
Builds on: Segment Anything Model AVSBench AV-SAM VGGish bottleneck adapters
Closest papers by meaning
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.92)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.91)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.89)
- Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language (CVPR 2024 · similarity 0.89)
- Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation (CVPR 2025 · similarity 0.89)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.