Research map / Vision-language and multimodal / Cross-modal learning
Revisiting Audio-Visual Segmentation with Vision-Centric Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Vision-Centric Transformer (VCT) for audio-visual segmentation that builds object queries from visual features (via a Prototype Prompted Query Generation module) and iteratively fetches audio and visual information, achieving state-of-the-art on AVSBench.
Problem
Audio-centric Transformers used in audio-visual segmentation suffer from perception ambiguity from mixed audio and weakened dense prediction due to loss of visual detail.
Approach
Derive object queries from visual regions rather than audio, prompt them with learnable audio-event prototypes trained via a prototype-audio contrastive loss, group pixel context with Gumbel-softmax hard assignment, and iteratively interact with audio and multi-scale visual features in a Mask2Former-style decoder.
Key results
- On the AVSS subset with PVT-v2 at 224x224, VCT reaches 44.7 MJ, surpassing AVSBias (Swin-B, 384x384) at 44.4 MJ.
- With Swin-B at 384x384, VCT achieves 51.2 MJ / 55.5 MF on AVSS, 86.2 MJ on MS3, and 67.6 MJ on S4 (per Table 1 layout).
- Ablation on AVSS with ResNet-50: vision-derived queries with audio prototypes reach 37.5 MJ vs 33.2 MJ for audio-derived queries baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VCT instead of AVSBias | AVSS, Swin-B · AVSS | MJ: 44.4 -> 51.2 | Helped |
| use VCT instead of AVSBias | S4, Swin-B · S4 | MJ: 83.3 -> 86.2 | Helped |
| use VCT instead of AVSBias | MS3, Swin-B · MS3 | MJ: 67.2 -> 67.6 | Helped |
| use VCT instead of AVSBias | MS3, Swin-B · MS3 | MF: 80.8 -> 81.4 | Helped |
| use vision-derived queries rather than audio-derived queries (ablation) | VCT on AVSS · AVSS | MJ: 33.2 -> 35.2 | Helped |
| add cross-attention to naive vision-derived queries (ablation) | VCT on AVSS · AVSS | MJ: 35.2 -> 35.8 | Helped |
| add Gumbel-softmax group-attention for pixel context grouping (ablation) | VCT on AVSS · AVSS | MJ: 35.8 -> 36.3 | Helped |
| add audio prototype prompting (ablation) | VCT on AVSS · AVSS | MJ: 36.3 -> 37.5 | Helped |
| add prototype-audio contrastive loss (ablation) | audio prototype prompting in VCT · AVSS | MJ: 36.3 -> 37.5 | Helped |
| use visual loss instead of no loss for audio prototype prompting (ablation) | VCT on AVSS · AVSS | MJ: 36.3 -> 36.5 | Helped |
| multiply audio and visual features (ablation) | VCT on AVSS · AVSS | MJ: 37.5 -> 33.9 | Hurt |
| concatenate audio and visual features (ablation) | VCT on AVSS · AVSS | MJ: 37.5 -> 35.3 | Hurt |
Surprising findings
- Randomly initialized audio prototypes without a loss give no gain over no prototypes, and visual-query contrastive loss gives only a small gain; the PAC loss is needed for the improvement.
Limitations
- Relies on Semantic-SAM priors and additional visual fusion, adding complexity
- Gains on S4 are modest where performance is plateaued
Experimental setup
Datasets: AVSBench S4 AVSBench MS3 AVSBench AVSS
Benchmarks: AVSBench S4 AVSBench MS3 AVSBench AVSS
Metrics: MJ (mean IoU) MF (F-score)
Baselines: TPAVI AQFormer ECMVAE CATR AVSC BAVS AVSBG GAVS AVSF UFE CAVP COMBO
Builds on: Mask2Former DETR GroupViT TPAVI COMBO
Closest papers by meaning
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.92)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.91)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.90)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.88)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.88)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.