Research map / Video / Video-language understanding
Benchmarking Audio Visual Segmentation for Long-Untrimmed Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LU-AVS, a large-scale benchmark of long untrimmed VGGSound videos with frame-level sounding-interval and mask/bounding-box annotations, and benchmarks adapted AVS, AVL, AVE and STVG methods plus a simple audio-visual baseline.
Problem
Existing audio-visual segmentation datasets use short trimmed clips with one mask per second, so they do not test temporal localization of sounding objects in long untrimmed videos where sound onset/offset and silent periods vary.
Approach
A long-untrimmed AVS dataset with per-frame mask annotations and bounding boxes for sounding objects plus start/end sounding times, and new metrics (m tIoU, m vIoU, m tF, m vF) that jointly assess temporal and spatial audible-object localization.
Key results
- Adapted AVS baselines TPAVI and ECMVAE reach at most 14.35% m tF and 6.73% m vF on LU-AVS, with none surpassing 15%.
- AVE methods achieve over 33% on all metrics, e.g. CMPAE reaching 40.67% m tIoU, but they do not localize spatially.
- The proposed mask-based strong baseline improves m vIoU from 6.03% (TPAVI) to 17.32%, and the bounding-box variant reaches 15.89% m vIoU, 8.88 points above STCAT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the mask-based LU-AVS baseline instead of TPAVI | LU-AVS audio-visual segmentation · LU-AVS | m vIoU: 6.03 -> 17.32 | Helped |
| Use the mask-based LU-AVS baseline instead of TPAVI | LU-AVS audio-visual segmentation · LU-AVS | m tIoU | Helped |
| Use the bounding-box-based LU-AVS baseline instead of EZLSL | LU-AVS audio-visual localization · LU-AVS | m vIoU: +8.96% | Helped |
| Use the bounding-box-based LU-AVS baseline instead of EZLSL | LU-AVS audio-visual localization · LU-AVS | m tIoU: +3.52% | Helped |
| Use the bounding-box-based LU-AVS baseline instead of STCAT | LU-AVS spatio-temporal grounding · LU-AVS | m vIoU: +8.88% | Helped |
| Adapt trimmed-video AVS methods to long-untrimmed videos | TPAVI and ECMVAE · LU-AVS | m vIoU | Hurt |
| Adapt AVL methods that do not account for silent segments to LU-AVS | AVL methods · LU-AVS | m vIoU | Hurt |
| Use AVE methods focused on temporal localization | AVE methods · LU-AVS | m vIoU: 40.10 | Helped |
| Replace the STVG text branch with an audio branch | STVG methods on LU-AVS · LU-AVS | m vIoU | Hurt |
| Use explicit audio-visual correlation rather than the STVG transformer-based multi-modal interaction | LU-AVS baseline · LU-AVS | m vIoU: +8.88% | Helped |
Surprising findings
- Adapted AVS methods designed for trimmed clips perform poorly on long untrimmed videos, with m vIoU around 5-6%.
- AVL methods trained on LU-AVS tend to overlook audible objects because silent segments make up a substantial share of the data.
Limitations
- Existing AVS, AVL, AVE and STVG methods were adapted rather than designed for untrimmed video, so the comparisons depend on those adaptations.
- Bounding-box annotations are provided for some categories that cannot be masked, so mask and box results are not directly comparable across the full set.
Experimental setup
Datasets: LU-AVS VGGSound AVSBench-Object AVSBench-Semantic VPO-SS VPO-MS
Benchmarks: LU-AVS
Metrics: m tIoU m vIoU m tF m vF
Baselines: TPAVI ECMVAE EZLSL LVS SLAVC SSPL AVEL CPSP JoMoLD CMPAE TubeDETR STCAT
Builds on: SAM XMem VGGSound AVSBench
Closest papers by meaning
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.89)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.89)
- Audio-Visual Segmentation via Unlabeled Frame Exploitation (CVPR 2024 · similarity 0.88)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.88)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.87)
- Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.