Research map / Video / Video-language understanding
STELLA: Continual Audio-Video Pre-training with SpatioTemporal Localized Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STELLA, a continual audio-video pre-training method that selects spatio-temporally aligned patches via cross-attention importance scores and replay-guided correlation scores, reducing forgetting and GPU memory.
Problem
Continual audio-video pre-training suffers from sparse spatio-temporal audio-video correlations and forgetting of previously learned multimodal correlations as the data distribution shifts over time.
Approach
Localized patch importance scoring via a lightweight audio-video matching (AVM) cross-attention module identifies strongly correlated patches, and replay-guided correlation assessment compares current queries against stored past queries to exclude patches tied to past correlations; patches are then sampled probabilistically and trained within the DER++ framework.
Key results
- STELLA+ reaches 15.44 average A->V retrieval on Continual-VS vs 12.52 for DER++, with lower forgetting (4.62 vs 5.68).
- On Continual-AS, STELLA+ improves average video-to-audio retrieval by 3.69 points over the baselines, as claimed in the paper text.
- STELLA uses about 17.45 GB GPU memory vs 30.95 GB for DER++, roughly a 45% reduction, while STELLA+ achieves 17.15 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STELLA instead of ESMER | Continual audio-video pre-training · Continual-VS audio-to-video zero-shot retrieval | Avg accuracy: 13.58 -> 14.16 | Helped |
| Use STELLA instead of ESMER | Continual audio-video pre-training · Continual-AS video-to-audio zero-shot retrieval | Avg accuracy: +3.69%p | Helped |
| Store selected patches in rehearsal memory (STELLA+) | STELLA · Continual-AS audio-to-video zero-shot retrieval | Avg accuracy: 12.35 -> 15.26 | Helped |
| Increase batch size by 1.66 times (STELLA++) | STELLA+ · Continual-VS audio-to-video zero-shot retrieval | Avg accuracy: 15.44 -> 17.01 | Helped |
| Add LPIS to the patch selection method (ablation) | STELLA built on DER++ · Continual-VS zero-shot retrieval (A-to-V) | Avg accuracy: 12.52 -> 13.44 | Helped |
| Add RCA to the patch selection method (ablation) | STELLA built on DER++ · Continual-VS zero-shot retrieval (A-to-V) | Avg accuracy: 12.52 -> 13.40 | Helped |
| Add LPIS and RCA together (ablation) | STELLA built on DER++ · Continual-VS zero-shot retrieval (A-to-V) | Avg accuracy: 12.52 -> 14.16 | Helped |
| Add LPIS and RCA together (ablation) | STELLA built on DER++ · Continual-VS zero-shot retrieval (V-to-A) | Avg accuracy: 11.91 -> 14.07 | Helped |
| Store selected patches in memory instead of raw data | STELLA · Audiovisual zero-shot retrieval | Avg accuracy: 14.16 -> 15.44; 14.07 -> 15.53; 12.35 -> 15.26; 11.29 -> 14.47 | Mixed |
| Reduce GPU memory consumption with STELLA+ | DER++ · Continual pre-training | GPU memory: 30.95 -> 17.15 | Helped |
Surprising findings
- Random patch selection with DER++-style replay (Random baseline) uses less memory than STELLA but gives lower retrieval scores, and RCA alone yields lower average retrieval but better forgetting.
Limitations
- Evaluated only on VGGSound and AudioSet-derived continual splits with a fixed task order; results vary with task ordering.
- Rehearsal memory still stores raw or selected video patches, raising privacy concerns noted in the impact statement.
Experimental setup
Datasets: VGGSound AudioSet MSR-VTT AVE AVSBench
Benchmarks: Continual-VS zero-shot retrieval Continual-AS zero-shot retrieval MSR-VTT audiovisual retrieval AVE audiovisual event classification AVSBench audiovisual segmentation AVE sound source localization
Metrics: average accuracy average forgetting R@1 R@5 R@10 MIoU GPU memory throughput modality gap
Baselines: Finetune ER MIR DER++ GMED CLS-ER LUMP ESMER MATS
Builds on: DER++ CAV MAE TVLT
Closest papers by meaning
- MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (CVPR 2025 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.86)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.86)
- JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation (ICLR 2026 · similarity 0.86)
- AVQACL: A Novel Benchmark for Audio-Visual Question Answering Continual Learning (CVPR 2025 · similarity 0.86)
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.