Research map / Continual learning / Continual fine-tuning of LLMs
Continual Multimodal Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates continual multimodal contrastive learning (CMCL) and proposes DNS, a replay-free dual-sided null-space gradient projection method that preserves prior modality-pair alignments while learning new pairs.
Problem
Continually adapting pretrained multimodal contrastive encoders (e.g., ImageBind, LanguageBind, UniBind) as new modality-pair data arrives, without forgetting previously learned cross-modal alignments (stability) while still learning new pairs (plasticity).
Approach
Derives gradient updates that are projected from both sides (dual-sided) onto null spaces of previous-step feature covariances, so the update for the new modality pair leaves prior alignment scores unchanged; the projection is computed via truncated SVD of uncentered feature covariances and is replay-free.
Key results
- On ImageBind classification, DNS reaches 52.52% Acc with BWT of -0.02, versus 50.96% for the best baseline C-FLAT.
- On LanguageBind classification, DNS reaches 64.07% Acc vs 60.15% for GEM, with BWT -0.09.
- DNS achieves the best R@1/R@5/R@10 across all three modality-binding backbones in retrieval; e.g., ImageBind R@10 40.89 vs 39.97 for C-FLAT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply dual-sided null-space gradient projection instead of C-FLAT | ImageBind · CMCL classification | Acc: 50.96 ± 0.34 -> 52.52 ± 0.23 | Helped |
| Apply dual-sided null-space gradient projection instead of CILA | ImageBind · CMCL classification | BWT_A: -3.20 ± 0.74 -> -0.02 ± 0.30 | Helped |
| Apply dual-sided null-space gradient projection instead of C-FLAT | ImageBind · CMCL retrieval | R@1: 9.92 ± 0.09 -> 10.38 ± 0.09 | Helped |
| Apply dual-sided null-space gradient projection instead of C-FLAT | ImageBind · CMCL retrieval | R@5: 28.90 ± 0.14 -> 29.53 ± 0.17 | Helped |
| Apply dual-sided null-space gradient projection instead of C-FLAT | ImageBind · CMCL retrieval | R@10: 39.97 ± 0.19 -> 40.89 ± 0.10 | Helped |
| Apply dual-sided null-space gradient projection instead of CILA | ImageBind · CMCL retrieval | BWT_R10: -1.28 ± 0.21 -> -1.07 ± 0.12 | Helped |
| Apply dual-sided null-space gradient projection instead of GEM | LanguageBind · CMCL classification | Acc: 60.15 ± 0.63 -> 64.07 ± 0.37 | Helped |
| Apply dual-sided null-space gradient projection instead of CILA | LanguageBind · CMCL classification | BWT_A: -2.63 ± 0.91 -> -0.09 ± 0.59 | Helped |
| Apply dual-sided null-space gradient projection instead of GEM | LanguageBind · CMCL retrieval | R@1: 8.32 ± 0.13 -> 8.71 ± 0.06 | Helped |
| Apply dual-sided null-space gradient projection instead of GEM | LanguageBind · CMCL retrieval | R@5: 28.92 ± 0.14 -> 29.81 ± 0.23 | Helped |
| Apply dual-sided null-space gradient projection instead of GEM | LanguageBind · CMCL retrieval | R@10: 41.24 ± 0.22 -> 42.44 ± 0.20 | Helped |
| Apply dual-sided null-space gradient projection instead of CILA | LanguageBind · CMCL retrieval | BWT_R10: -1.09 ± 0.40 -> -3.00 ± 0.21 | Hurt |
Surprising findings
- Replay-based GEM achieves higher LanguageBind accuracy than DER, DER++ and Co2L in the reported table, while its ImageBind accuracy drops, showing backbone-dependent baseline behavior.
Limitations
- Theoretical bounds are derived under linear encoder-adapter assumptions (linear learner W per modality on frozen priors).
- Relies on uncentered feature covariance and truncated SVD with a minimum eigenvalue hyperparameter that must be tuned per backbone.
Experimental setup
Datasets: UCF101 ESC50 NYUDv2 VGGSound-S Clotho TVL LLVIP
Benchmarks: UCF101 ESC50 NYUDv2 VGGSound-S Clotho TVL LLVIP
Metrics: Acc Recall@1 Recall@5 Recall@10 BWT
Baselines: Vanilla GEM DER DER++ EWC Co2L C-FLAT CILA
Builds on: Adam-NSCL CLIP ImageBind LanguageBind UniBind
Closest papers by meaning
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.90)
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.89)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.88)
- QUEST: Quadruple Multimodal Contrastive Learning with Constraints and Self-Penalization (NeurIPS 2024 · similarity 0.88)
- A Practitioner's Guide to Real-World Continual Multimodal Pretraining (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.