Research map / Image generation / Text-to-image diffusion
MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MuLan, a lightweight (<20M parameter) language adapter that bridges a frozen image-centered multilingual text encoder (InternVL-LLaMA) with frozen diffusion models, trained only on English image-text pairs to enable T2I generation in 110+ languages.
Problem
Existing text-to-image diffusion models support mainly English and a few Latin-script languages; native multilingual T2I models need costly curated multilingual image data, and translation-based pipelines suffer latency and translation errors.
Approach
Use a multilingual text encoder pre-trained with image-text contrastive alignment on noisy web data (InternVL-LLaMA), freeze it and the diffusion model, and train only a small language adapter on English image-text pairs so multilingual prompts map into the diffusion conditioning space; the adapter stays plug-and-play with LoRA, ControlNet, LCM, and IP-Adapter.
Key results
- MuLan-PixArt reaches average CLIP similarity 39.5 across XM12 languages versus 39.6 for English, and MuLan-SD15 reaches 37.7 average.
- Average CLIP score on five non-English languages: InternVL-LLaMA-based adapter 37.8 vs. 33.2-36.1 for other text encoders (MultiLang-CLIP, AltClip-m18, Mul-OpenCLIP).
- Training MuLan-SD15 takes about 12 hours on 8 A100 GPUs, versus AltDiffusion requiring at least 19x64 GPU-days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt PixArt-α with MuLan instead of using Google Translate | PixArt-α · XM12 | SIM: 39.7 vs. 39.5 | Hurt |
| Adapt PixArt-α with MuLan instead of using Google Translate | PixArt-α · XM12 | FID: 13.7 vs. 11.8 | Hurt |
| Adapt Stable Diffusion 1.5 with MuLan instead of using multilingual prompts as-is | Stable Diffusion 1.5 · XM12 | SIM: 29.1 -> 37.7 | Helped |
| Use MuLan with InternVL-LLaMA rather than specialized models | Stable Diffusion 1.5 · XM3600 Chinese | SIM: 36.84 -> 37.84 | Helped |
| Use MuLan with InternVL-LLaMA rather than JapaneseSDXL | Stable Diffusion 1.5 · XM3600 Japanese | SIM: 38.00 -> 38.01 | Helped |
| Use image-centered alignment rather than language-centered alignment | MuLan with Stable Diffusion 1.5 · XM12, five non-English languages | avg SIM: 33.2 to 36.1; 33.3 to 37.8 | Helped |
| Use a multilingual encoder without multilingual semantic alignment (ablation) | MuLan with Stable Diffusion 1.5 · XM12, non-English prompts | avg SIM | Hurt |
| Reduce training data from 17M to 8.5M samples (ablation) | MuLan with Stable Diffusion 1.5 · XM12, seven-language average | SIM: 35.80 -> 35.46 | Hurt |
| Reduce training data from 17M to 4.25M samples (ablation) | MuLan with Stable Diffusion 1.5 · XM12, seven-language average | SIM: 35.80 -> 35.40 | Hurt |
| Reduce training data from 17M to 2.1M samples (ablation) | MuLan with Stable Diffusion 1.5 · XM12, seven-language average | SIM: 35.80 -> 35.23 | Hurt |
| Reduce training data from 17M to 1M samples (ablation) | MuLan with Stable Diffusion 1.5 · XM12, seven-language average | SIM: 35.80 -> 35.17 | Hurt |
| Reduce training data from 17M to 250k samples (ablation) | MuLan with Stable Diffusion 1.5 · XM12, seven-language average | SIM: 35.80 -> 35.13 | Hurt |
Surprising findings
- Text encoders trained only on multilingual text corpora (LLaMA2-7B, mT5-xl, XLM-R Large) fail to understand non-English prompts when only the adapter is trained on English data.
- Reducing the English training data to 1/1024 (16k samples) causes a sharp drop, but performance is largely stable down to 1/256 (63k samples).
Limitations
- Trained on paired data, which may introduce bias and distribution shift relative to original models.
- Language-centered alignment alternatives perform worse, and the approach depends on an image-aligned multilingual encoder.
- FID on XM12 is worse than AltDiffusion for the PixArt variant.
Experimental setup
Datasets: LAION-EN PixArt dataset COCO2014 XM3600
Benchmarks: XM3600 (XM12 subset) COCO2014 validation (85 languages)
Metrics: CLIP score (SIM) FID LAION aesthetic score VQAScore
Baselines: AltDiffusion GlueGen PEA-Diffusion Taiyi JapaneseSDXL Google Translate + SD1.5 NLLB + PixArt-alpha
Builds on: Stable Diffusion PixArt-alpha InternVL-LLaMA min-SNR weighting
Closest papers by meaning
- AltDiffusion: A Multilingual Text-to-Image Diffusion Model (AAAI 2024 · similarity 0.92)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.89)
- MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis (AAAI 2025 · similarity 0.88)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.88)
- Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.