Research map / Image generation / Text-to-image diffusion
AltDiffusion: A Multilingual Text-to-Image Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AltDiffusion, a multilingual text-to-image diffusion model supporting 18 languages by distilling a multilingual XLM-R text encoder into OpenCLIP and training a Stable Diffusion UNet in two stages, plus two new benchmarks (MG-18, MC-18).
Problem
English-only text-to-image diffusion models like Stable Diffusion serve non-English users poorly, and translation-based prompting loses information, especially for culture-specific concepts; existing multilingual T2I models cover only a few languages.
Approach
Distill a multilingual XLM-R text encoder to align with OpenCLIP's text embedding space, plug it into a pretrained English-only SD model, and train it in two stages (cross-attention K/V alignment at 256px on multilingual LAION, then full UNet fine-tuning on aesthetic data at 512px) to extend language support to 18 languages.
Key results
- On MG-18, AltDiffusion beats translation-based SD v2.1 on CLIP Sim for all 18 languages (e.g., English 0.324 vs 0.291), with comparable FID and IS.
- Against Taiyi-Bilingual and Japanese SD, AD improves CLIP Sim by 24.1% (English), 26.8% (Chinese) and 28.1% (Japanese).
- In human evaluation on MC-18, AD wins the total score in all 17 reported languages except none lost; e.g., Chinese 0.769 vs 0.231 for SD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AltDiffusion instead of translation-based Stable Diffusion | MG-18 multilingual prompts · MG-18, all 18 languages | CLIP Sim | Helped |
| Use AltDiffusion instead of translation-based Stable Diffusion | MG-18 multilingual prompts · MG-18, general image quality | FID and IS | Mixed |
| Use AltDiffusion instead of multilingual baseline diffusion models | T2I generation · MG-18 English, Chinese, and Japanese | CLIP Sim: +24.1%, +26.8%, and +28.1% | Helped |
| Use AltDiffusion instead of translation-based Stable Diffusion | culture-specific multilingual generation · MC-18, all 18 languages | Total preference score | Helped |
| Add the quality-improvement training stage after concept alignment (ablation) | AltDiffusion · Generated image quality | Image quality | Helped |
Surprising findings
- On MG-18 FID, translation-based SD v2.1 is better than AD in most languages, so the gains are mainly in CLIP Sim rather than image quality.
- Hindi and Arabic human-eval results show SD nearly tied with AD on culture consistency (e.g., Hindi 4.824 vs 4.784).
Limitations
- Image quality (FID, IS) is comparable to or slightly below English SD rather than clearly better.
- Evaluation relies on a small culture-specific benchmark (50 prompts per language) and a CLIP-based metric that may be biased toward its own encoder family.
Experimental setup
Datasets: LAION2B-en LAION2B-multi LAION Aesthetics XM3600 WIT MS-COCO 2017 validation
Benchmarks: MG-18 MC-18 MMS-COCO
Metrics: FID IS CLIP Sim human culture consistency score human text-image consistency score
Baselines: Stable Diffusion v2.1 with NLLB-3B translation Taiyi-Bilingual Taiyi-CN Japanese Stable Diffusion
Builds on: AltCLIP Stable Diffusion OpenCLIP XLM-R classifier-free guidance
Closest papers by meaning
- MuLan: Adapting Multilingual Diffusion Models for Hundreds of Languages with Negligible Cost (ICML 2025 · similarity 0.92)
- Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities (AAAI 2025 · similarity 0.91)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.90)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.