Research map / Continual learning / Continual fine-tuning of LLMs
CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sequentially fine-tune the full encoder on VQAv2 -> NLVR2 -> SNLI-VE -> VCR | ViLT · upstream VL tasks | knowledge transfer: 0.13 -> -5.09 | Hurt |
| use Adapters (new modules per task, frozen encoder) instead of sequential fine-tuning | ViLT · VCR upstream | task score: 59.47 -> 61.18 | Helped |
| freeze the entire encoder | ViLT · VCR upstream | task score: 61.31 -> 41.90 | Hurt |
| add Experience Replay to sequential fine-tuning | SeqFT ViLT · forgetting across upstream tasks | forgetting | Helped |
| add EWC regularization to sequential fine-tuning | SeqFT ViLT · forgetting across upstream tasks | forgetting | No clear effect |
| add EWC to sequential fine-tuning | SeqFT ViLT · SNLI-VE upstream | task score: 74.89 -> 74.38 | Hurt |
| upstream continual learning before low-shot fine-tuning | ViLT · downstream low-shot multimodal tasks | low-shot transfer | Hurt |
| include VCR in upstream task order (ablation) | ViLT SeqFT · upstream transfer and forgetting | knowledge transfer / forgetting | Hurt |
| use VAuLT (BERT language input) instead of ViLT encoder (ablation) | low-shot language tasks · SST-2 and other language tasks | accuracy | Helped |
| freeze bottom 9 layers during upstream CL (ablation) | ViLT · downstream low-shot vision tasks | low-shot transfer | Helped |
Closest papers by meaning
- Don't Forget Why You Started: Tackling Dual Forgetting in Vision-Language Continual Learning (ICML 2026 · similarity 0.91)
- Stabilizing Zero-Shot Prediction: A Novel Antidote to Forgetting in Continual Vision-Language Tasks (NeurIPS 2024 · similarity 0.89)
- Enhancing Visual Continual Learning with Language-Guided Supervision (CVPR 2024 · similarity 0.89)
- A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.89)
- Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning (NeurIPS 2026 · similarity 0.89)
- Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.