Research map / Vision-language and multimodal / CLIP and zero-shot
GMAIL: Generative Modality Alignment for generated Image Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GMAIL, which treats Stable Diffusion-generated images as a separate modality, aligns a LoRA-fine-tuned CLIP image encoder on generated images to the real-image CLIP encoder via a cross-modality contrastive loss, and uses it to train captioning and retrieval models.
Problem
Training vision-language models on generated images directly causes modality discrepancies between generated and real images, leading to degraded performance and mode collapse on real data.
Approach
Keep a real-image CLIP encoder unchanged and fine-tune a separate LoRA-adapted encoder on generated images, aligning its embeddings with real-image embeddings of the same caption through a contrastive cross-modality loss; the aligned encoder is then used to train downstream VLMs on generated data while real images go through the original encoder at inference.
Key results
- On COCO captioning, ClipCap + GMAIL improves B@4 from 32.15 to 38.12 and CIDEr from 108.35 to 119.53 over the ClipCap baseline.
- CLIP + GMAIL on COCO text-to-image R@1 rises from 32.7 to 37.5, and on Flickr30k text-to-image R@1 from 24.7 to 30.2.
- Scaling from COCO to CC12M synthetic data raises Flickr30k image-to-text R@1 from 47.1 to 50.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Gen-Real alignment | ClipCap on COCO image captioning · COCO | CIDEr: 108.35 -> 119.53 | Helped |
| add Gen-Real alignment | CLIP zero-shot image-to-text retrieval · COCO | Recall@1: 51.8 -> 56.8 | Helped |
| add Gen-Real alignment | CLIP zero-shot image-to-text retrieval · Flickr30k | Recall@1: 44.1 -> 47.1 | Helped |
| add Gen-Real alignment | CLIP zero-shot classification · DTD | top-1 accuracy: 55.20 -> 65.26 | Helped |
| add Gen-Real alignment | CLIP long-caption image-to-text retrieval · ShareGPT4V | Recall@1: 78.2 -> 85.2 | Helped |
| add Gen-Real alignment (ablation) | CLIP image captioning fine-tuned on generated images · COCO | CIDEr: 115.35 -> 119.53 | Helped |
| use LoRA rank 2 instead of rank 4 (ablation) | CLIP image captioning fine-tuned on generated images · COCO | CIDEr: 119.53 -> 117.62 | Hurt |
| use LoRA rank 6 instead of rank 4 (ablation) | CLIP image captioning fine-tuned on generated images · COCO | CIDEr: 119.53 -> 119.12 | Hurt |
| replace LoRA rank 4 with full fine-tuning (ablation) | CLIP image captioning fine-tuned on generated images · COCO | CIDEr: 119.53 -> 118.95 | Hurt |
| add dual projection and alignment to synthetic-image fine-tuning (ablation) | CLIP image captioning · COCO | CIDEr: 115.35 -> 119.53 | Helped |
| add dual projection and alignment to real-image fine-tuning (ablation) | CLIP image captioning · COCO | CIDEr: 119.78 -> 119.95 | Helped |
| increase generated training data from COCO to CC12M (ablation) | GMAIL zero-shot image-to-text retrieval · Flickr30k | Recall@1: 47.1 -> 50.9 | Helped |
Surprising findings
- Synthetic-data training with GMAIL alignment reportedly matches or slightly exceeds training on real COCO images (B@4 38.37 vs 38.24 for real-only with the same dual projection setup).
Limitations
- Relies on quality of the generative model; poor generators may reduce alignment effectiveness.
- Introduces additional computation for the dual-projection and alignment setup.
Experimental setup
Datasets: COCO CC3M CC12M ShareGPT4V DTD Stanford Cars SUN397 Food 101 FGVC Aircraft Oxford Pets Caltech 101 ImageNet
Benchmarks: COCO Flickr30k ShareGPT4V DTD Stanford Cars SUN397 Food 101 FGVC Aircraft Oxford Pets Caltech 101 ImageNet ScienceQA
Metrics: BLEU@4 METEOR CIDEr SPICE ROUGE-L WMD Recall@1 Recall@5 Recall@10 top-1 accuracy
Baselines: CLIP Long-CLIP ClipCap IFCap LLaVA Llama3 SynCLR SigLIP Task2Sim
Builds on: CLIP LoRA ClipCap Long-CLIP
Closest papers by meaning
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.89)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)
- Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.88)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.88)
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models (ICLR 2024 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.