Research map / Video / Video segmentation and tracking
Audio-Visual Segmentation via Unlabeled Frame Exploitation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UFE, a framework that exploits unlabeled video frames for audio-visual segmentation by feeding optical flow from neighboring frames as motion input and using a teacher-student weak-to-strong consistency scheme on distant frames.
Problem
Audio-visual segmentation models trained on sparsely annotated videos underutilize unlabeled frames, with global temporal modeling giving only marginal gains from them.
Approach
Divides unlabeled frames into neighboring frames (NFs, used for optical-flow motion cues as extra model input) and distant frames (DFs, used as semantic augmentations via teacher-student pseudo-labeling with weak-to-strong consistency), a versatile plug-in for existing AVS architectures.
Key results
- On AVSBench S4 with PVT backbone, the AVSegFormer-based method reaches 83.15 mIoU, and with ResNet it reaches 78.96 mIoU, both new state-of-the-art claims.
- Applied to TPAVI (ResNet), the framework improves S4 mIoU from 72.79 to 78.15 and MS3 mIoU from 47.88 to 54.08.
- Ablation on TPAVI with ResNet shows NF alone gives 76.17 and DF alone gives 77.43 mIoU on S4, versus 72.80 baseline; combined gives 78.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add UFE (neighboring-frame motion cues and distant-frame self-training) | TPAVI (ResNet) · AVSBench-S4 | mIoU (MJ): 72.79 -> 78.15 (+5.36) | Helped |
| Add UFE (neighboring-frame motion cues and distant-frame self-training) | TPAVI (PVT) · AVSBench-S4 | mIoU (MJ): 78.74 -> 82.49 (+3.75) | Helped |
| Add UFE (neighboring-frame motion cues and distant-frame self-training) | AVSegFormer (ResNet) · AVSBench-MS3 | mIoU (MJ): 49.53 -> 55.88 (+6.35) | Helped |
| Add UFE (neighboring-frame motion cues and distant-frame self-training) | AVSegFormer (PVT) · AVSBench-MS3 | mIoU (MJ): 58.36 -> 61.95 (+3.59) | Helped |
| Add neighboring frames (motion guidance) to TPAVI (ablation) | TPAVI (ResNet) · AVSBench-S4 | mIoU (MJ): 72.80 -> 76.17 | Helped |
| Add distant frames (self-training) to TPAVI (ablation) | TPAVI (ResNet) · AVSBench-S4 | mIoU (MJ): 72.80 -> 77.43 | Helped |
| Add both neighboring and distant frames to TPAVI (ablation) | TPAVI (ResNet) · AVSBench-S4 | mIoU (MJ): 72.80 -> 78.15 | Helped |
| Set burn-in duration to 5 epochs instead of 10 (ablation) | TPAVI with teacher-student training · AVSBench-S4 | mIoU (MJ): 78.15 -> 77.72 | Hurt |
| Set burn-in duration to 20 epochs instead of 10 (ablation) | TPAVI with teacher-student training · AVSBench-S4 | mIoU (MJ): 78.15 -> 77.50 | Hurt |
| Set burn-in duration to 30 epochs instead of 10 (ablation) | TPAVI with teacher-student training · AVSBench-S4 | mIoU (MJ): 78.15 -> 77.48 | Hurt |
| Use unsupervised-loss weight λ=0.1 instead of λ=0.5 (ablation) | TPAVI with teacher-student training · AVSBench-S4 | mIoU (MJ): 78.15 -> 77.00 | Hurt |
| Use unsupervised-loss weight λ=0.2 instead of λ=0.5 (ablation) | TPAVI with teacher-student training · AVSBench-S4 | mIoU (MJ): 78.15 -> 77.62 | Hurt |
Surprising findings
- Global temporal modeling with unlabeled frames yields only marginal gains (0.14 mIoU with ResNet) over training with labeled frames only, indicating underutilization.
Limitations
- Optical flow requires computing flow from neighboring frames (Farneback), adding preprocessing cost.
- Gains reported are on a single benchmark family (AVSBench) with two baselines.
Experimental setup
Datasets: AVSBench S4 AVSBench MS3 VGGSound
Benchmarks: AVSBench S4 AVSBench MS3
Metrics: mIoU F-score
Baselines: TPAVI AVSegFormer AVSC CATR AuTR ECMVAE SAMA-AVS
Builds on: TPAVI AVSegFormer Mean Teacher-style weak-to-strong consistency
Closest papers by meaning
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.89)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.89)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.88)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.88)
- AVSegFormer: Audio-Visual Segmentation with Transformer (AAAI 2024 · similarity 0.88)
- Benchmarking Audio Visual Segmentation for Long-Untrimmed Videos (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.