Research map / Video / Video-language understanding
Audio-Visual Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces audio-visual instance segmentation (AVIS), the AVISeg benchmark (926 long videos, 94K masks), and AVISM, a query-based baseline with frame-level localizer and window-attention video-level tracker, evaluated against VIS/AVSS methods and multi-modal LLMs.
Problem
Existing audio-visual segmentation work handles only binary or semantic masks on short trimmed clips and cannot distinguish or track individual sounding instances of the same category in long videos.
Approach
A frame-level audio-conditioned query localizer compresses per-frame scenes into object tokens, and a video-level tracker uses window-based self-attention with audio cross-attention over these tokens to track sounding instances across long videos.
Key results
- AVISM with ResNet-50 reaches 42.78 FSLA, 61.73 HOTA and 40.57 mAP on AVISeg, beating the best prior method COMBO (39.49 FSLA, 57.39 HOTA, 37.84 mAP).
- Swin-L backbone with IN+COCO pre-training improves AVISM to 52.49 FSLA, 71.13 HOTA and 53.46 mAP.
- Zero-shot multi-modal LLM pipelines perform poorly, with the best (AL-Ref-SAM 2) reaching 18.55 FSLA and 15.84 mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AVISM's audio-visual localization and tracking design | AVISM versus the strongest reported related-task baseline · AVISeg test | FSLA: 39.49 -> 42.78 | Helped |
| add frame-level audio-visual fusion (ablation) | AVISM · AVISeg test | FSLA: 38.04 -> 39.68 | Helped |
| add video-level audio-visual fusion to frame-level fusion (ablation) | AVISM · AVISeg test | FSLA: 39.68 -> 42.78 | Helped |
| increase tracker local window size from 3 to 12 (ablation) | AVISM · AVISeg test | HOTA: 61.13 -> 62.82 | Helped |
| add COCO pretraining to the visual backbone (ablation) | AVISM with ResNet-50 · AVISeg test | FSLA: 42.78 -> 44.42 | Helped |
| add OVIS fine-tuning after IN+COCO pretraining (ablation) | AVISM with ResNet-50 · AVISeg test | FSLA: 44.42 -> 43.68 | Hurt |
| replace ResNet-50 with ResNet-101 (ablation) | AVISM with IN+COCO pretraining · AVISeg test | FSLA: 44.42 -> 45.06 | Helped |
| replace ResNet-50 with Swin-L (ablation) | AVISM with IN+COCO pretraining · AVISeg test | FSLA: 44.42 -> 52.49 | Helped |
Surprising findings
- Adding OVIS pre-training raised mAP but slightly lowered FSLA, attributed to segmenting non-sounding objects.
- Zero-shot multi-modal large models perform far below specialized baselines on instance-level sound localization and temporal perception.
Limitations
- Evaluation of multi-modal LLMs is zero-shot and limited to a few models.
- Video-level tracker's max frame capacity decreases with larger window size, trading off tracking accuracy and long-sequence processing.
Experimental setup
Datasets: AVISeg COCO OVIS ImageNet MUSIC-AVQA YTVIS
Benchmarks: AVISeg
Metrics: FSLA HOTA mAP AssA DetA
Baselines: Mask2Former-VIS TeViT SeqFormer VITA DAVIS LBVQ AVSegFormer COMBO Sam4AVS BuboGPT PG-Video-LLaVA AL-Ref-SAM 2
Builds on: Mask2Former VITA IFC DETR VGGish
Closest papers by meaning
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.92)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.90)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.90)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.89)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.89)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.