Research map / Continual learning / Continual fine-tuning of LLMs
Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bisecle proposes a neurobiology-inspired parameter-efficient continual learning method for VideoQA on a LLaMA-Adapter backbone, combining multi-directional supervision (question and video prediction) with contrastive prompt learning over task type embeddings.
Problem
Continual learning for video question answering with frozen large VLM/LLM backbones suffers from catastrophic forgetting and update conflicts in the small set of trainable parameters.
Approach
Adds auxiliary bidirectional objectives (predict question from video and answer, predict video from question and answer) to mimic hippocampal rapid binding, and a contrastive loss that aligns question-reweighted shared prompts with learnable task-type embeddings to mimic pattern separation, without task-specific parameters at test time.
Key results
- On NExT-QA, Bisecle reaches 62.37% Acc and 5.34 Fog versus 46.58% Acc and 13.83 Fog for the LLaMA-Adapter backbone and 55.14% Acc for the strongest baseline ColPro.
- On DramaQA, Bisecle reaches 71.49% Acc and 10.37 Fog, outperforming ColPro (71.24%, 12.64 Fog).
- On STAR, Bisecle reaches 52.16% Acc and 7.60 Fog, the best among compared methods, with ProgPrompt at 51.07% Acc.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ColPro with Bisecle | LLaMA-2-7B continual VideoQA · DramaQA | accuracy: 71.24 -> 71.49 | Helped |
| Replace ColPro with Bisecle | LLaMA-2-7B continual VideoQA · DramaQA | forgetting: 12.64 -> 10.37 | Helped |
| Add question prediction supervision LQ (ablation) | Bisecle · NExT-QA | accuracy: 46.58 -> 61.13 | Helped |
| Add video prediction supervision LV (ablation) | Bisecle · NExT-QA | accuracy: 46.58 -> 53.94 | Helped |
| Add contrastive prompt loss LP (ablation) | Bisecle · NExT-QA | accuracy: 46.58 -> 55.82 | Helped |
| Add contrastive prompt loss LP to LQ + LV (ablation) | Bisecle · NExT-QA | accuracy: 59.78 -> 62.37 | Helped |
| Add question prediction supervision LQ to LV + LP (ablation) | Bisecle · NExT-QA | accuracy: 51.95 -> 62.37 | Helped |
| Add video prediction supervision LV to LQ + LP (ablation) | Bisecle · NExT-QA | accuracy: 61.38 -> 62.37 | Helped |
| Use 8 rather than 32 prompt-injection layers (ablation) | Bisecle · NExT-QA | accuracy: 62.37 -> 53.86 | Hurt |
| Use 16 rather than 32 prompt-injection layers (ablation) | Bisecle · NExT-QA | accuracy: 62.37 -> 58.18 | Hurt |
| Use 24 rather than 32 prompt-injection layers (ablation) | Bisecle · NExT-QA | accuracy: 62.37 -> 57.30 | Hurt |
| Use class-mean question tokens instead of learnable task type embeddings (ablation) | Bisecle contrastive prompt learning · NExT-QA | accuracy: 62.37 -> 58.96 | Hurt |
Surprising findings
- On STAR, ColPro (48.67% Acc) underperforms several other baselines such as ProgPrompt (51.07%), despite being a closer counterpart.
- Frontier models (Gemini 2.0 Flash, GPT-4o) were evaluated zero-shot, where forgetting metrics are not meaningful; the authors note that the backbone's temporal reasoning weaknesses persist in these models.
Limitations
- Assumes task boundaries are known during training; task-free continual learning not addressed.
- Evaluated only on VideoQA with three datasets and a single LLaMA-Adapter backbone family.
- Gains on some task orders are inconsistent, with forgetting higher than the backbone for some orders in the appendix table.
Experimental setup
Datasets: NExT-QA DramaQA STAR
Benchmarks: NExT-QA DramaQA STAR
Metrics: Avg. Acc Avg. Fog
Baselines: LLaMA-Adapter L2P DualPrompt LAE DAM ProgPrompt ColPro
Builds on: LLaMA-Adapter ColPro L2P DualPrompt
Closest papers by meaning
- ViLCo-Bench: VIdeo Language COntinual learning Benchmark (NeurIPS 2024 · similarity 0.88)
- ReWind: Understanding Long Videos with Instructed Learnable Memory (CVPR 2025 · similarity 0.87)
- VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models (CVPR 2026 · similarity 0.87)
- $\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation (ICML 2025 · similarity 0.87)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.