Research map / Vision-language and multimodal / Cross-modal learning
Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces few-shot incremental learning to audio-visual semantic segmentation and proposes FINGER, with class-centric foreground aggregation and a dual-tower knowledge transfer design plus proxy calibration.
Problem
Audio-visual semantic segmentation models cannot incorporate new sound-producing classes from few annotated samples without forgetting old classes, and old classes in incremental samples are unlabeled background.
Approach
Trains without a background class by learning one proxy per foreground class with one-vs-rest binary masks (background inferred as complement), uses separate audio and visual towers for knowledge transfer before cross-modal fusion, and freezes the base model while a proxy calibration module learned via episodic training transfers base-class knowledge to novel proxies.
Key results
- On AVSBench-Semantic, FINGER reaches 47.1 mIoU and 54.2 F-score, versus 36.7 mIoU and 42.0 F-score for AVSegFormer.
- In the 5-shot incremental setting, FINGER achieves 19.2 mIoU-N and 26.7 mIoU-H, compared with 11.4 mIoU-N and 17.1 mIoU-H for AVSegFormer+PIFS.
- Ablation shows dual-tower proxy aggregation improves 5-shot mIoU-N from 7.9 (single tower) to 19.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FINGER instead of AVSegFormer | AVSBench-Semantic · AVSBench-Semantic | mIoU: 36.7 -> 47.1 | Helped |
| use FINGER instead of AVSegFormer | AVSBench-Semantic · AVSBench-Semantic | F-score: 42.0 -> 54.2 | Helped |
| use FINGER instead of AVSegFormer+PIFS | FSIL-AVSS · FSIL-AVSS 5-shot incremental | mIoU-H: 17.1 -> 26.7 | Helped |
| replace conventional attention with proxy aggregation (ablation) | FINGER · AVSBench-Semantic | mIoU: 37.0 -> 47.1 | Helped |
| use a dual tower instead of a single tower (ablation) | FINGER with proxy aggregation · AVSBench-Semantic | mIoU: 38.5 -> 47.1 | Helped |
| add proxy calibration (ablation) | FINGER · FSIL-AVSS 5-shot incremental | mIoU-N: 2.7 -> 14.8 | Helped |
| train proxy calibration with episode training (ablation) | FINGER · FSIL-AVSS 5-shot incremental | mIoU-H: 21.4 -> 26.7 | Helped |
Surprising findings
- Existing AVSS model AVSegFormer nearly collapses on novel classes in the incremental setting (0.1 mIoU-N in 1-shot), indicating severe catastrophic forgetting and poor rapid learning.
Limitations
- Incremental performance on novel classes remains low in absolute terms (mIoU-N 5.9 in 1-shot).
- Evaluated only on AVSBench-Semantic with a fixed split of 50 base and 20 novel classes.
Experimental setup
Datasets: AVSBench-Semantic
Benchmarks: AVSBench-Semantic FSIL-AVSS benchmark
Metrics: mIoU F-score mIoU-B mIoU-N mIoU-H
Baselines: 3DC AOT AVSBench CATR BAVS AVSegFormer AVSegFormer+MiB AVSegFormer+PIFS
Builds on: AVSBench AVSegFormer MiB PIFS
Closest papers by meaning
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.90)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.89)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.89)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.87)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.87)
- QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.