Research map / Continual learning / Domain and prompt-based
Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Few-Shot Audio-Visual Class-Incremental Learning (FS-AVCIL) and proposes a ViT-based audio-visual adapter with temporal-residual prompting plus an exemplar-free temporal prompt regularization to reduce forgetting and overfitting.
Problem
Continually learning new audio-visual classes from only a few labeled examples per class, where models must both fuse audio and visual modalities under limited data and avoid catastrophic forgetting of cross-modal correlations.
Approach
Builds on LAVISH's audio-visual adapters by inserting temporal-residual prompting (1D temporal convolution over concatenated audio-visual tokens feeding latent tokens via cross-modal attention) and regularizes the temporal prompts across sessions with a cosine-similarity loss so no exemplars are stored.
Key results
- On AVE protocol-3 the method reaches 66.92% final accuracy versus 54.83% for LAVISH-based fine-tuning and 57.30% average-accuracy-style baseline AV-CIL at 63.78% final accuracy.
- On AVE protocol-1 the method reports 78.76% average accuracy, above AV-CIL at 73.03%.
- Ablation on AVE protocol-3 shows TRP-AVA and TPR together improve final accuracy from 30.26% (Ft-ViT) to 66.92%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full TRP-AVA and TPR framework instead of AV-CIL | ViT-Base · AVE protocol-3 | Average Acc.: 63.78 -> 66.92 | Helped |
| Add TRP-AVA (ablation) | FS-AVCIL · AVE protocol-3 | Average Acc.: 38.63 -> 55.74 | Helped |
| Add temporal prompt regularization (ablation) | FS-AVCIL with TRP-AVA · AVE protocol-3 | Average Acc.: 55.74 -> 66.92 | Helped |
| Add TRP-AVA and temporal prompt regularization (ablation) | FS-AVCIL · AVE protocol-3 | Average Acc.: 25.83 -> 66.92 | Helped |
| Use two latent tokens rather than four or six (ablation) | TRP-AVA · AVE FS-AVCIL | classification performance | Helped |
| Keep the residual operation on latent tokens (ablation) | TRP-AVA · AVE FS-AVCIL | classification performance | Helped |
Surprising findings
- Few-shot audio-visual results show performance fluctuation rather than monotonic decline on longer task sequences, attributed to modality bias.
Limitations
- Performance fluctuates on longer task sequences (protocol-3/4), attributed to modality bias.
- Evaluated only on two audio-visual datasets (AVE and Kinetics-Sounds).
Experimental setup
Datasets: AVE Kinetics-Sounds
Benchmarks: AVE Kinetics-Sounds
Metrics: final accuracy performance dropping rate average accuracy
Baselines: Ft-ViT G-Blend MBT LAVISH LWF EWC iCaRL DER L2P AV-CIL CIGN
Builds on: LAVISH AV-CIL Adapters ViT
Closest papers by meaning
- Few-Shot Incremental Learning via Foreground Aggregation and Knowledge Transfer for Audio-Visual Semantic Segmentation (AAAI 2025 · similarity 0.90)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.88)
- Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds (CVPR 2025 · similarity 0.88)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.88)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.88)
- Aligned Better, Listen Better for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.