Research map / Vision-language and multimodal / CLIP and zero-shot
Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MpM, a training paradigm that pivots multimodal learning from English through a multilingual LLM so English-only image-text training yields zero-shot Chinese (and other-language) image-to-text and text-to-image models, realized as VisCPM.
Problem
Large multimodal models are mostly English-only because non-English languages lack large-scale high-quality image-text data; the paper asks whether multimodal alignment learned in English transfers zero-shot to other languages via a multilingual LLM.
Approach
Use a strong bilingual/multilingual LLM as a pivot: train visual modules only on English image-text pairs (pretraining plus instruction tuning, or cross-attention for diffusion), so that the frozen or shared LLM's multilingual representations transfer multimodal alignment to the target language, with a small amount of translated instruction data to fix response language.
Key results
- VisCPM-Chat achieves 90.9 average on the Chinese LLaVA Test Set, above Qwen-VL-Chat (88.2) and other Chinese-English baselines, despite no Chinese multimodal data in pretraining; VisCPM-Chat+ reaches 92.5.
- VisCPM-Paint reaches a Chinese zero-shot FID of 9.6 on MSCOCO with native plus translated Chinese fine-tuning, versus 10.9 for English-only pretraining and 15.1 for native Chinese only (per ablation).
- In human evaluation on Chinese DrawBench (174 prompts), VisCPM-Paint receives the strongest preference across Overall, Alignment, and Fidelity, with more than 40% of its images favored in each aspect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Chinese image-text pairs during pretraining (ablation) | VisCPM-Chat · LLaVA Test Set, Chinese | GPT-4 AVG score: 90.9 -> 92.5 | Helped |
| add Chinese image-text pairs during pretraining (ablation) | VisCPM-Chat · LLaVA Test Set, English | GPT-4 AVG score: 81.4 -> 81.5 | No clear effect |
| use English rather than native Chinese image-text data for pretraining (ablation) | image-to-text model with Chinese instruction tuning · LLaVA Test Set, Chinese | GPT-4 AVG score: 85.5 -> 86.3 | Helped |
| use bilingual rather than monolingual Chinese instruction tuning (ablation) | image-to-text model with English pretraining · LLaVA Test Set, Chinese | GPT-4 AVG score: 86.3 -> 88.0 | Helped |
| add native Chinese image-text data during pretraining (ablation) | image-to-text model with bilingual instruction tuning · LLaVA Test Set, Chinese | GPT-4 AVG score: 88.0 -> 87.8 | No clear effect |
| add translated Chinese image-text data during pretraining (ablation) | image-to-text model with bilingual instruction tuning · LLaVA Test Set, Chinese | GPT-4 AVG score: 87.8 -> 88.2 | No clear effect |
| use English rather than native Chinese image-text data for pretraining (ablation) | VisCPM-Paint · MSCOCO validation, Chinese | FID: 15.1 -> 10.9 | Helped |
| fine-tune on unfiltered native Chinese image-text pairs (ablation) | VisCPM-Paint · MSCOCO validation, Chinese | FID: 10.9 -> 12.8 | Hurt |
| fine-tune on filtered native Chinese image-text pairs (ablation) | VisCPM-Paint · MSCOCO validation, Chinese | FID: 10.9 -> 10.7 | No clear effect |
| add translated Chinese image-text pairs to fine-tuning (ablation) | VisCPM-Paint · MSCOCO validation, Chinese | FID: 10.7 -> 9.6 | Helped |
Surprising findings
- Instruction tuning on English only yields quasi-zero-shot behavior: the model answers Chinese questions correctly but mostly in English, and only 0.1% translated Chinese instruction data suffices to get over 80% response-language accuracy.
- Native Chinese Wukong image-text pairs were found to be largely poorly matched (low CLIP scores), and adding unfiltered native Chinese data hurt text-to-image FID (10.9 to 12.8).
Limitations
- Evaluation of Chinese relies on GPT-4 judging on a small LLaVA test set of 90 instances and a 400-instance UniMM-Bench, with translated benchmarks.
- Text-to-image Chinese results rely on a small human-evaluation set of 174 prompts; English FID is only comparable to, not better than, strong English models.
- Multilingual extension to five European languages is evaluated only on the LLaVA test set and IGLUE, with no native-speaker human evaluation reported.
Experimental setup
Datasets: LAION-2B Laion-COCO CC3M CC12M COCO Visual Genome Wukong Zero LLaVA-Instruct-150K UniMM-Chat M3IT IGLUE
Benchmarks: LLaVA Test Set UniMM-Bench MSCOCO validation (zero-shot FID) Chinese DrawBench IGLUE xGQA IGLUE xVNLI
Metrics: GPT-4 rating score zero-shot FID CLIP score human preference rate
Baselines: MiniGPT-4 InstructBLIP LLaVA mPLUG-Owl VisualGLM Ziya-Visual Qwen-VL-Chat AltDiffusion TaiyiDiffusion
Builds on: LLaVA Stable Diffusion CPM-Bee Muffin BLIP-2
Closest papers by meaning
- Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages (ICLR 2025 · similarity 0.88)
- AltDiffusion: A Multilingual Text-to-Image Diffusion Model (AAAI 2024 · similarity 0.88)
- DiM$^3$: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Merging (NeurIPS 2026 · similarity 0.88)
- MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost (ICML 2025 · similarity 0.88)
- AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.88)
- LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.