Research map / Image generation / Text-to-image diffusion
Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ITIT, a unified image-text encoder with separate image and text decoders trained with cycle-consistency losses (T2I2T and I2T2I) so that unpaired image-only and text-only data can supplement a small paired set for generative vision-language training.
Problem
Generative vision-language models (text-to-image and image-to-text) need huge paired image-text corpora, which are costly to annotate or noisy when web-scraped; unpaired images and texts go unused.
Approach
A joint image-text encoder with disjoint autoregressive text decoder and parallel masked-token image decoder, trained on unpaired data by synthesizing the missing modality and reconstructing the original input (text-image-text and image-text-image cycles), with straight-through/Gumbel gradient estimation through argmax outputs and gradient truncation to keep cycle training tractable.
Key results
- ITIT-H with 3M paired (CC3M) and 110M unpaired (WebLI) reaches FID 10.4 and zero-shot CIDEr 38.2 on COCO, beating several models trained on far more paired data for T2I.
- With 1.2M paired and 396.8M unpaired Shutterstock samples, ITIT achieves 19.2 FID and 21.0 CIDEr, close to 19.0 FID and 22.2 CIDEr from 398M paired samples.
- Full T2I2T plus I2T2I cycle training on CC3M plus WebLI unpaired reaches FID 14.3 and CIDEr 31.1, improving on paired-only (15.7 FID, 23.5 CIDEr) in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add full T2I2T and I2T2I cycle losses using unpaired WebLI images and text (ablation) | ITIT-B trained on paired CC3M · MS-COCO | FID: 15.7 -> 14.3 | Helped |
| Add full T2I2T and I2T2I cycle losses using unpaired WebLI images and text (ablation) | ITIT-B trained on paired CC3M · MS-COCO | CIDEr: 23.5 -> 31.1 | Helped |
| Use low-quality WebLI as paired rather than unpaired data (ablation) | ITIT-B trained on paired CC3M · MS-COCO | CIDEr: 23.5 -> 20.7 | Hurt |
| Add unpaired WebLI through T2T and I2I training without cycle losses (ablation) | ITIT-B trained on paired CC3M · MS-COCO | CIDEr: 23.5 -> 28.5 | Helped |
| Replace full cycle losses with half-cycle losses (ablation) | ITIT-B trained on paired CC3M and unpaired WebLI · MS-COCO | FID: 14.3 -> 14.5 | Hurt |
| Replace full cycle losses with half-cycle losses (ablation) | ITIT-B trained on paired CC3M and unpaired WebLI · MS-COCO | CIDEr: 31.1 -> 30.5 | Hurt |
| Add full T2I2T and I2T2I cycle losses without additional data (ablation) | ITIT-B trained with CC3M · MS-COCO | FID: 15.7 -> 15.4 | Helped |
| Add full T2I2T and I2T2I cycle losses without additional data (ablation) | ITIT-B trained with CC3M · MS-COCO | CIDEr: 23.5 -> 24.4 | Helped |
| Use Gumbel softmax instead of straight-through softmax for T2I2T gradient estimation (ablation) | ITIT T2I2T cycle · COCO Karpathy split | CIDEr (zs): +0.3 | Helped |
| Use Gumbel softmax for I2T2I gradient estimation (ablation) | ITIT I2T2I cycle · MS-COCO | generation performance | No clear effect |
| Add the full T2I2T cycle (ablation) | ITIT-B with unpaired WebLI text · MS-COCO | FID and CIDEr | Mixed |
| Add the full I2T2I cycle (ablation) | ITIT-B with unpaired WebLI images · MS-COCO | FID and CIDEr | Mixed |
Surprising findings
- Adding unpaired data via cycle loss scales similarly to adding paired data, while adding low-quality WebLI as paired data harms captioning severely.
- Naive text-to-text or image-to-image unpaired baselines without cycle loss improve results less than the full cycle approach.
Limitations
- Gains are evaluated mainly on MS-COCO; scaling to larger unpaired data and model sizes is left to future work.
- Model comparisons use different paired data mixtures, so some reported gains rest on re-implemented baselines (e.g., Muse re-implementation).
Experimental setup
Datasets: CC3M WebLI Shutterstock MS-COCO Karpathy split COCO Captions
Benchmarks: MS-COCO COCO Karpathy split captioning COCO Captions 30K FID
Metrics: FID CIDEr
Baselines: Stable Diffusion GLIDE DALL-E 2 Parti Muse BLIP SimVLM GIT CoBIT CM3Leon
Builds on: GIT Muse MAGE VQGAN CoBIT
Closest papers by meaning
- MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis (AAAI 2025 · similarity 0.90)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)
- Learning Visual Generative Priors without Text (CVPR 2025 · similarity 0.89)
- Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (NeurIPS 2024 · similarity 0.89)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- TLDR: Token-Level Detective Reward Model for Large Vision Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.