Research map / Vision-language and multimodal / CLIP and zero-shot
Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends CLIP's language coverage to 36 languages via continual language learning, training only an expandable token embedding layer, and proposes TEIR (identically distributed token embedding initialization plus lexical-statistics-based regularization) to reduce catastrophic forgetting.
Problem
Multilingual vision-language pre-trained models are costly to retrain jointly when new languages arrive, and extending dual-stream VL-PTMs like CLIP to new languages incrementally suffers from catastrophic forgetting caused by covariate shift and lexical overlap.
Approach
Freeze all CLIP components and train only an expandable token embedding layer under cross-modal (InfoNCE) and cross-lingual (MSE between English and foreign text features) objectives; initialize new token embeddings from the same distribution as prior ones and scale weight decay and gradients per token by how often it appeared in earlier tasks.
Key results
- TEIR improves CLL-CLIP text-to-image AR on XM3600 from 17.6 to 24.3 (+6.7), with consistent gains across oEWC, ER, DER, MLA, P-Tuning, LoRA, DualPrompt and CodaPrompt.
- On Multi30K zero-shot, CLL-CLIP with TEIR averages 64.5 vs. 53.8 without TEIR, and MLA with TEIR reaches 70.7 vs. 58.4 for MLA.
- Joint-learning CLL-CLIP reaches 37.1 text-to-image AR on XM3600, which the authors treat as the upper bound for continual models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TEIR to CLL-CLIP | CLL-CLIP · XM3600 out-of-domain, text-to-image | average Recall@1: 17.6 -> 24.3 (+6.7) | Helped |
| Add TEIR to continual learning methods | oEWC, ER, DER, MLA, P-Tuning, LoRA, DualPrompt, and CodaPrompt · MSCOCO₃₆ and XM3600 | average Recall@1: absolute improvements ranging from 3.7% to 10.2% in text-to-image AR on XM3600 | Helped |
| Use identical-distribution initialization for new token embeddings (ablation) | CLL-CLIP · MSCOCO₃₆, image-to-text | average Recall@1: 29.6 -> 32.4 | Helped |
| Use gradient regularization when updating token embeddings (ablation) | CLL-CLIP · MSCOCO₃₆, image-to-text | average Recall@1: 29.6 -> 31.9 | Helped |
| Use weight-decay regularization when updating token embeddings (ablation) | CLL-CLIP · MSCOCO₃₆, image-to-text | average Recall@1: 29.6 -> 33.3 | Helped |
| Combine gradient and weight-decay regularization (ablation) | CLL-CLIP · MSCOCO₃₆, image-to-text | average Recall@1: 31.9 -> 37.2 | Helped |
| Combine identical-distribution initialization with gradient and weight-decay regularization (ablation) | CLL-CLIP · XM3600, text-to-image | average Recall@1: 22.8 -> 24.3 | Helped |
| Build an oracle shared vocabulary from all languages (ablation) | CLL-CLIP with TEIR · XM3600, text-to-image | average Recall@1: 24.3 -> 27.1 | Helped |
| Add TEIR | CLL-CLIP · Multi30K, average across languages | average retrieval score: 53.8 -> 64.5 | Helped |
| Add TEIR | MLA · Multi30K, Czech | average retrieval score: 64.6 -> 74.1 | Helped |
| Add CC3M training data | CLL model evaluated on Hebrew · XM3600, Hebrew, text-to-image | Recall@1: 44.3 vs. 46.7 | Helped |
| Train only token embeddings instead of the full model (ablation) | CLL-CLIP · Continual language learning | knowledge retention | Helped |
Surprising findings
- Using an oracle vocabulary built from all languages improves results further (setting 7 vs 6), suggesting vocabulary substitution is a bottleneck.
Limitations
- Evaluated only on CLIP-style dual-stream image-text retrieval, not encoder-decoder models or generation tasks.
- TEIR requires current-task text statistics to compute the regularization, limiting boundary-free continual learning.
- A gap remains between continual and joint learning performance.
Experimental setup
Datasets: MSCOCO XM3600 Multi30K CC3M
Benchmarks: MSCOCO36 XM3600 Multi30K
Metrics: Recall@1 average recall (AR) forgetting (F)
Baselines: oEWC ER DER MLA P-Tuning LoRA DualPrompt CodaPrompt M-CLIP PaLI M3P UC2
Builds on: CLIP OpenCLIP Elastic Weight Consolidation Dark Experience Replay DualPrompt
Closest papers by meaning
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.89)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.89)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.