Research map / Vision-language and multimodal / Cross-modal learning
AVSegFormer: Audio-Visual Segmentation with Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVSegFormer, a transformer-based audio-visual segmentation framework with a dense audio-visual channel-attention mixer and a sparse query-based audio-visual decoder, achieving state-of-the-art on AVSBench.
Problem
Audio-visual segmentation (locating and segmenting sounding objects in video from audio), where existing fusion methods struggle with evolving audio, multiple simultaneous sources, and multiple coexisting objects.
Approach
Combines a dense audio-conditioned channel-attention mixer that reweights visual mask features with a Mask2Former-style sparse decoder whose audio-conditioned queries separate sound sources, giving bidirectional audio-visual conditioning; an auxiliary mixing loss supervises the mixer.
Key results
- AVSegFormer-PVTv2 reaches 82.06 mIoU on S4 and 58.36 mIoU on MS3, outperforming AVSBench-PVTv2 (78.74 and 54.00).
- On AVSS, AVSegFormer-PVTv2 obtains 36.66 mIoU and 42.0 F-score versus 29.77 mIoU and 35.2 F-score for AVSBench-PVTv2.
- With 512x512 input, AVSegFormer-PVTv2* reaches 83.06 mIoU (S4), 61.33 mIoU (MS3), and 37.31 mIoU (AVSS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 300 queries instead of 1 (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 79.6 -> 83.1 | Helped |
| Use 100 queries instead of 1 (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 79.6 -> 81.4 | Helped |
| Use 200 queries instead of 1 (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 79.6 -> 82.3 | Helped |
| Add learnable queries alongside audio queries (ablation) | AVSegFormer-PVTv2 · MS3 | mIoU: 58.5 -> 61.3 | Helped |
| Add learnable queries alongside audio queries (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 82.7 -> 83.1 | Helped |
| Add a channel-attention audio-visual mixer instead of no mixer (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 81.4 -> 83.1 | Helped |
| Use the cross-attention mixer (CRA) instead of no mixer (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 81.4 -> 82.7 | Helped |
| Use channel-attention mixer (CHA) instead of cross-attention mixer (CRA) (ablation) | AVSegFormer-PVTv2 · MS3 | mIoU: 59.8 -> 61.3 | Helped |
| Add auxiliary mixing loss (ablation) | AVSegFormer-PVTv2 · S4 | mIoU: 81.3 -> 83.1 | Helped |
| Add auxiliary mixing loss (ablation) | AVSegFormer-PVTv2 · MS3 | mIoU: 59.6 -> 61.3 | Helped |
| Use AVSegFormer-PVTv2 instead of AVSBench-PVTv2 | Audio-visual segmentation models · S4 | mIoU: +3.32 | Helped |
| Use AVSegFormer-PVTv2 instead of AVSBench-PVTv2 | Audio-visual segmentation models · MS3 | mIoU: +1.65 | Helped |
Surprising findings
- A cross-attention audio-visual mixer performed worse than the channel-attention mixer, which the authors adopted after the initial design fell short.
Limitations
- Performance on AVSS remains low in absolute terms (best 36.66 mIoU with PVTv2 at standard resolution).
- Training details beyond those stated are deferred to supplementary materials.
Experimental setup
Datasets: AVSBench-Object AVSBench-Semantic ImageNet-1K AudioSet
Benchmarks: AVSBench S4 AVSBench MS3 AVSBench AVSS
Metrics: mIoU F-score
Baselines: AVSBench-R50 AVSBench-PVTv2 DiffusionAVS LVS MSSL 3DC SST AOT iGAN LGVT
Builds on: AVSBench Mask2Former DETR VGGish
Closest papers by meaning
- Revisiting Audio-Visual Segmentation with Vision-Centric Transformer (CVPR 2025 · similarity 0.92)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.92)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.90)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.90)
- QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition (CVPR 2024 · similarity 0.89)
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.