Research map / Continual learning / Continual fine-tuning of LLMs
Continual Vision-Language Retrieval via Dynamic Knowledge Rectification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Knowledge Rectification (DKR), which filters and rectifies incorrect old-model affinity knowledge in distillation so CLIP can be continually adapted to streaming image-text retrieval datasets with less forgetting.
Problem
Continual vision-language retrieval (CVLR): adapting CLIP to streams of image-text datasets without catastrophic forgetting of old retrieval performance, where class labels are unavailable for class-incremental methods.
Approach
Distills text-image affinity matrices from the old CLIP model but dynamically rectifies incorrect old affinities: diagonal entries are replaced by new-model affinities where only the new model is correct, and by paired ground-truth labels where neither model retrieves correctly, with off-diagonal entries rescaled accordingly.
Key results
- Under Setting-1 (five sequential datasets), DKR reaches average R@1 of 43.3%, versus 43.3 for ... and 42.0 for the strongest prior baseline Mod-X in the average column of Table 1 (DKR average R@1 43.3 vs Mod-X 41.9).
- On MS-COCO, DKR outperforms Mod-X by 1.7% R@1, 2.2% R@5 and 1.8% R@10 after sequential training.
- Ablation shows rectification with new knowledge adds 1.4 average R@1 and rectification with paired knowledge adds 0.7, combined giving 2.1 over the base distillation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DKR instead of the strongest listed baseline | CLIP ViT-B/32 continual retrieval · Setting-1, average over five datasets | R@1: 42.0 -> 43.3 | Helped |
| Add Rectification with New Knowledge (RNK) (ablation) | DKR Base · Setting-1, five datasets | average R@1: 41.2 -> 42.6 | Helped |
| Add Rectification with Paired Knowledge (RPK) (ablation) | DKR Base · Setting-1, five datasets | average R@1: 41.2 -> 41.9 | Helped |
| Add RPK to RNK (ablation) | DKR with RNK · Setting-1, five datasets | average R@1: 42.6 -> 43.3 | Helped |
| Remove RNK's diagonal-affinity enhancement in Eq. 8 (ablation) | DKR with RNK and RPK · Setting-1, five datasets | average R@1: 43.3 -> 42.9 | Hurt |
| Increase the distillation weight λ from 0.1 to 1 (ablation) | DKR · Setting-1, five datasets | average R@1: 42.15 -> 43.34 | Helped |
| Increase the distillation weight λ from 1 to 10 (ablation) | DKR · Setting-1, five datasets | average R@1: 43.34 -> 42.84 | Hurt |
Surprising findings
- On the EC-split setting, DKR's Flickr30K and MS-COCO image-to-text retrieval exceeds zero-shot CLIP even though these old datasets are not trained on directly.
Limitations
- Relies on known paired ground-truth labels for the rectification step, which assumes labeled image-text pairs are available in the stream.
- Rectification with new knowledge is ineffective or harmful when the new model cannot yet output correct affinities, e.g., for datasets far from the old ones.
Experimental setup
Datasets: MS-COCO Flickr30K IAPR TC-12 ECommerce-T2I RSICD
Benchmarks: MS-COCO Flickr30K IAPR TC-12 ECommerce-T2I RSICD
Metrics: R@1 R@5 R@10
Baselines: Joint Zero-Shot CLIP SFT EWC LwF ERL AFC Mod-X
Builds on: CLIP Mod-X Elastic Weight Consolidation Learning without Forgetting
Closest papers by meaning
- Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters (CVPR 2024 · similarity 0.88)
- Unified Knowledge Maintenance Pruning and Progressive Recovery with Weight Recalling for Large Vision-Language Models (AAAI 2025 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
- Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective (NeurIPS 2024 · similarity 0.87)
- Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval (NeurIPS 2025 · similarity 0.87)
- Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.