Research map / Continual learning / Class-incremental
Continual Audio-Visual Sound Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces continual audio-visual sound separation and ContAV-Sep, which uses a cross-modal similarity distillation constraint plus output distillation and a small memory to reduce catastrophic forgetting on MUSIC-21.
Problem
Audio-visual sound separation models trained sequentially on new sound classes suffer catastrophic forgetting of previously learned classes, and existing continual learning methods do not preserve cross-modal associations needed for separation.
Approach
A Cross-modal Similarity Distillation Constraint (CrossSDC) that combines instance-level and class-level contrastive cross-modal similarity losses, with distillation of similarities from old models on memory samples, added to output mask distillation on top of an iQuery-style separator.
Key results
- ContAV-Sep with iQuery reaches 7.33 SDR, 13.55 SIR, 13.01 SAR on MUSIC-21, versus 6.86/13.13/12.31 for the best baseline (AV-CIL with memory).
- Without memory, iQuery fine-tuning gets 3.46 SDR while the upper bound reaches 10.36 SDR.
- Increasing memory from 1 to 30 samples per old class raises ContAV-Sep SDR from 7.33 to 10.09.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply ContAV-Sep with iQuery instead of the strongest listed continual-learning baseline | iQuery · MUSIC-21 | SDR: 7.03 -> 7.33 | Helped |
| apply ContAV-Sep with iQuery instead of the strongest listed continual-learning baseline | iQuery · MUSIC-21 | SIR: 13.30 -> 13.55 | Helped |
| apply ContAV-Sep with iQuery instead of the strongest listed continual-learning baseline | iQuery · MUSIC-21 | SAR: 12.60 -> 13.01 | Helped |
| apply ContAV-Sep instead of the strongest listed baseline | iQuery · AVE | SDR: 2.53 -> 2.72 | Helped |
| apply ContAV-Sep instead of the strongest listed baseline | iQuery · AVE | SIR: 6.64 -> 7.32 | Helped |
| apply ContAV-Sep instead of the strongest listed baseline | iQuery · AVE | SAR: 12.83 -> 9.86 | Hurt |
| apply ContAV-Sep instead of the strongest listed baseline | iQuery · VGGSound | SDR: 4.71 -> 4.90 | Helped |
| apply ContAV-Sep instead of the strongest listed baseline | iQuery · VGGSound | SIR: 8.89 -> 9.25 | Helped |
| apply ContAV-Sep instead of the strongest listed baseline | iQuery · VGGSound | SAR: 12.84 -> 11.73 | Hurt |
| add CrossSDC to output distillation (ablation) | ContAV-Sep · MUSIC-21 | SDR: 6.32 -> 7.33 | Helped |
| add instance-aware CrossSDC to output distillation (ablation) | ContAV-Sep · MUSIC-21 | SDR: 6.32 -> 6.01 | Hurt |
| add class-aware CrossSDC to output distillation (ablation) | ContAV-Sep · MUSIC-21 | SDR: 6.32 -> 6.86 | Helped |
Surprising findings
- Even a single memory sample per old class substantially boosts all baselines, which the authors attribute to mixing samples to create many training pairs.
- Baseline continual methods do not improve old classes much, and the method may block old classes from acquiring new knowledge in later tasks.
Limitations
- Method mainly preserves old knowledge and may prevent old classes from acquiring new knowledge in later tasks.
- Object-based separators depend on object detectors and can miss sounding objects.
- Main experiments limited to MUSIC-21 with a small 20-class setting of 1040 videos.
Experimental setup
Datasets: MUSIC-21 AVE VGGSound
Benchmarks: MUSIC-21 AVE VGGSound
Metrics: SDR SIR SAR
Baselines: Fine-tuning LwF EWC PLOP EWF AV-CIL Upper Bound
Builds on: iQuery Co-Separation AV-CIL PLOP LwF
Closest papers by meaning
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.88)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.87)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.87)
- Mixtures of Experts for Audio-Visual Learning (NeurIPS 2024 · similarity 0.87)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.87)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.