Research map / Detection and segmentation / Open-set and few-shot detection
Improving Audio-Visual Segmentation with Bidirectional Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVS-BG, an audio-visual segmentation model with bidirectional generation (visual-to-audio reconstruction consistency) and an implicit correlation-pyramid motion module, reaching new SOTA on AVSBench.
Problem
Audio-visual segmentation (AVS) must pixel-segment sounding objects in video, but existing methods under-model cross-modal relationships between object appearance and sound and struggle with motion captured by optical flow.
Approach
A visual-to-audio projection reconstructs audio features from (masked) visual features and enforces audio-visual-audio consistency with the original audio embedding, combined with an implicit visual correlation pyramid between adjacent frames that models motion without flow supervision.
Key results
- Achieves 81.71 mIoU and 0.904 F-score on AVSBench S4, up from 78.74 for AVSBench (PVT).
- Achieves 55.10 mIoU and 0.668 F-score on MS3 from scratch, up from 54.00; 58.58 mIoU when pretrained on S4.
- Adding bidirectional generation to the original AVSBench raises S4 mIoU from 78.74 to 80.64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add audio features and the audio-visual fused module (ablation) | AVS framework · AVSBench S4, MS3 from scratch, and MS3 pretrained on Single-Source | mIoU: 80.16 -> 80.82; 51.85 -> 54.35; 55.63 -> 56.51 | Helped |
| Add the motion correlation module (ablation) | AVS framework with audio · AVSBench S4, MS3 from scratch, and MS3 pretrained on Single-Source | mIoU: 80.82 -> 81.26; 54.35 -> 54.91; 56.51 -> 58.33 | Helped |
| Add bidirectional generation (ablation) | AVSBench baseline · AVSBench S4, MS3 from scratch, and MS3 pretrained on Single-Source | mIoU: 78.74 -> 80.64; 54.00 -> 54.78; 57.34 -> 59.04 | Helped |
| Add bidirectional generation (ablation) | Ours without motion · AVSBench S4, MS3 from scratch, and MS3 pretrained on Single-Source | mIoU: 80.82 -> 81.85; 54.35 -> 53.49; 56.51 -> 58.81 | Mixed |
| Add bidirectional generation (ablation) | Ours with motion · AVSBench S4, MS3 from scratch, and MS3 pretrained on Single-Source | mIoU: 81.26 -> 81.71; 54.91 -> 55.10; 58.33 -> 58.58 | Mixed |
| Set consistency-loss η to 1.0 rather than 0.1, 0.5, or 1.5 (ablation) | AVS framework · AVSBench S4 and MS3 from scratch | mIoU: 81.40; 81.61; 81.71; 81.23; 53.07; 54.35; 55.10; 54.18 | Helped |
| Use PVT-v2 instead of ResNet50 as backbone (ablation) | Ours · AVSBench S4 and MS3 | mIoU: 74.13 -> 81.71; 44.95 -> 55.10; 49.58 -> 58.58 | Mixed |
| Pretrain on the Single-Source subset before MS3 (ablation) | Ours (PVT-v2) · AVSBench MS3 | mIoU: 55.10 -> 58.58 | Helped |
| Replace TPAVI and reduce neck channels from 256 to 128 | Ours compared with AVSBench · AVSBench inference | Time (ms): 43.55 -> 27.48 | Helped |
Surprising findings
- Bidirectional generation gives a larger gain when the motion module is removed (e.g., S4 80.82 to 81.85), and the MS3 from-scratch mIoU drops slightly (54.35 to 53.49) in that configuration.
Limitations
- Gains on MS3 are modest (e.g., 55.10 vs 54.00 from scratch) and the ResNet50 variant underperforms on MS3 from scratch compared with AVSBench (44.95 vs 47.88).
- Training uses small-scale AVSBench data; evaluation limited to a single benchmark.
Experimental setup
Datasets: AVSBench ImageNet AudioSet
Benchmarks: AVSBench S4 AVSBench MS3
Metrics: mIoU F-score
Baselines: AVSBench LVS MSSL 3DC SST iGAN LGVT
Builds on: AVSBench PVT-v2 VGGish TPAVI
Closest papers by meaning
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.92)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.92)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.91)
- Revisiting Audio-Visual Segmentation with Vision-Centric Transformer (CVPR 2025 · similarity 0.91)
- Unsupervised Audio-Visual Segmentation with Modality Alignment (AAAI 2025 · similarity 0.90)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.