Research map / Video / Video-language understanding
ViLCo-Bench: VIdeo Language COntinual learning Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ViLCo-Bench, the first video-language continual learning benchmark built from Ego4D (moment query, natural language query, visual query tasks), plus a memory-efficient baseline (ViLCo) using SSL with narrations and a dual short/long-term memory.
Problem
Video-language continual learning lacks a standardized benchmark; existing CL benchmarks focus on class-incremental classification with short videos and single modality, not long egocentric video with non-classification multimodal queries.
Approach
Defines query-incremental learning over Ego4D episodic memory tasks (MQ, NLQ, VQ) with ~10-minute videos, and proposes ViLCo, which combines self-supervised contrastive learning on video narrations with a prompt-pool episodic memory and short-term replay buffer.
Key results
- On ViLCo-Bench MQ, ViLCo reaches Avg R@1 (IoU=0.5) of 26.24% with BwF 2.9, outperforming iCaRL (23.66%, BwF 4.6) and EWC (12.51%, BwF 24.2).
- On NLQ (13 tasks), ViLCo achieves Avg R@1 mean 7.85 and BwF 10.60 versus Naive 4.83 (BwF 48.76) and MAS 5.63 (BwF 35.92).
- On VQ, ViLCo gets avg tAP25 17.85% and BwF 23.77 versus MAS 12.13% (BwF 47.60) and EWC 11.48%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViLCo instead of iCaRL | Moment Query continual learning · Moment Query | Avg R@1 (IoU=0.5): 23.66 -> 26.24 | Helped |
| Use ViLCo instead of MAS | Natural Language Query continual learning · Natural Language Query | Avg R@1 (IoU=0.5): 4.22 -> 6.21 | Helped |
| Use ViLCo instead of MAS | Visual Query continual learning · Visual Query | Avg tAP25: 12.13 -> 17.85 | Helped |
| Add episodic memory (ablation) | ViLCo without EM · Moment Query | Avg R@1 (IoU=0.5): 25.86 -> 26.24 | Helped |
| Add self-supervised learning (ablation) | ViLCo without SSL · Moment Query | Avg R@1 (IoU=0.5): 24.49 -> 26.24 | Helped |
| Add episodic memory and self-supervised learning (ablation) | Naive · Moment Query | Avg R@1 (IoU=0.5): 17.58 -> 26.24 | Helped |
| Add InternVideo features (ablation) | ViLCo with EgoVLP-v2 · Moment Query | Avg R@1 (IoU=0.5): 26.24 -> 33.53 | Helped |
| Add Slowfast features to EgoVLP-v2 + InternVideo (ablation) | ViLCo with EgoVLP-v2 + InternVideo · Moment Query | Avg R@1 (IoU=0.5): 33.53 -> 29.75 | Hurt |
| Add Omnivore features to EgoVLP-v2 + InternVideo + Slowfast (ablation) | ViLCo with EgoVLP-v2 + InternVideo + Slowfast · Moment Query | Avg R@1 (IoU=0.5): 29.75 -> 28.18 | Hurt |
| Replace EgoVLP-v2 with ViViT (ablation) | ViLCo · Moment Query | Avg R@1 (IoU=0.5): 26.24 -> 35.82 | Helped |
| Replace EgoVLP-v2 with Timersformer (ablation) | ViLCo · Moment Query | Avg R@1 (IoU=0.5): 26.24 -> 22.82 | Hurt |
| Replace EgoVLP-v2 with X3D (ablation) | ViLCo · Moment Query | Avg R@1 (IoU=0.5): 26.24 -> 23.01 | Hurt |
Surprising findings
- Replay-based BiC performs far worse than other methods on MQ (Avg R@1 IoU=0.5 of 3.39%) despite its low BwF of 1.4.
- Adding more visual backbones is not always better: EgoVLP-v2 + InternVideo improves results, but adding SlowFast and Omnivore degrades R@1 and increases BwF.
Limitations
- Fixed backbones and task-specific heads; does not generalize to arbitrary tasks such as VQA or video captioning.
- Heavy reliance on annotated data streams; dataset is half from the US, risking societal and cultural bias.
- Replay-based methods are sensitive to task order, with performance varying across orderings.
Experimental setup
Datasets: Ego4D UCF101
Benchmarks: ViLCo-Bench-MQ ViLCo-Bench-NLQ ViLCo-Bench-VQ
Metrics: Avg R@1 (IoU=0.3, 0.5) Avg R@5 Backward Forgetting (BwF) temporal AP (tAP25) stAP25 avg recall success rate
Baselines: EWC MAS iCaRL BiC Naive Upper-Bound Lower-Bound
Builds on: EgoVLP EWC iCaRL BiC MAS
Closest papers by meaning
- Bisecle: Binding and Separation in Continual Learning for Video Language Understanding (NeurIPS 2025 · similarity 0.88)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.87)
- Scalable Language Model with Generalized Continual Learning (ICLR 2024 · similarity 0.87)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.86)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.