Research map / Vision-language and multimodal / CLIP and zero-shot
Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a DDPM on CLIP text embeddings and applies a reverse diffusion bridge at an intermediate step to map image embeddings toward the text distribution, enabling zero-shot captioning with a GPT-2 prefix decoder trained only on text.
Problem
The CLIP modality gap between image and text embeddings limits zero-shot image captioning, since decoders trained on text embeddings transfer poorly to image embeddings.
Approach
Train a diffusion model only on centered, normalized text embeddings, then inject image embeddings at an intermediate reverse-diffusion step so they are progressively denoised into text-like embeddings; augment decoder training with diffusion-perturbed text embeddings to handle the resulting variability.
Key results
- On MSCOCO in-domain zero-shot, Diffusion Bridge reaches CIDEr 96.4 and BLEU@4 28.7, state-of-the-art over listed baselines (C3 93.3 CIDEr).
- Paired image-text cosine similarity rises from 0.306 to 0.606 after diffusion bridging.
- On Flickr30K in-domain, CIDEr 53.0 is competitive but below DeCap (56.7), which uses a memory bank.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply diffusion-bridging to vision embeddings at inference (ablation) | Plain Embed on MSCOCO captioning · MSCOCO test | CIDEr: 25.2 -> 88.8 | Helped |
| add diffusion-augmented text embeddings to diffusion-bridging (ablation) | Diffusion Bridge on MSCOCO captioning · MSCOCO test | CIDEr: 88.8 -> 95.7 | Helped |
| increase the inference step from 0 to 200 (ablation) | Diffusion Bridge on MSCOCO captioning · MSCOCO test | CIDEr: 25.2 -> 65.9 | Helped |
| increase the inference step from 200 to 400 (ablation) | Diffusion Bridge on MSCOCO captioning · MSCOCO test | CIDEr: 65.9 -> 83.3 | Helped |
| increase the inference step from 400 to 600 (ablation) | Diffusion Bridge on MSCOCO captioning · MSCOCO test | CIDEr: 83.3 -> 88.8 | Helped |
| increase the inference step from 600 to 800 (ablation) | Diffusion Bridge on MSCOCO captioning · MSCOCO test | CIDEr: 88.8 -> 86.1 | Hurt |
| use Diffusion Bridge instead of C3 | zero-shot captioning · MSCOCO | CIDEr: 93.3 -> 96.4 | Helped |
| use Diffusion Bridge instead of DeCap | zero-shot captioning · Flickr30K | CIDEr: 56.7 -> 53.0 | Hurt |
| use Diffusion Bridge instead of MeaCap | cross-domain captioning · MSCOCO → Flickr30K | CIDEr: 40.3 -> 38.5 | Hurt |
| use Diffusion Bridge instead of ViECap | cross-domain captioning · Flickr30K → MSCOCO | CIDEr: 54.2 -> 47.1 | Hurt |
Surprising findings
- Projector-based similarity boosts unpaired similarity too (hallucination risk), whereas diffusion bridging keeps paired/unpaired selectivity.
Limitations
- Flickr30K performance lags memory-based DeCap, attributed to smaller training set limiting the diffusion model's learned distribution.
- Cross-domain Flickr30K->MSCOCO CIDEr and SPICE trail ViECap and MeaCap.
- Performance degrades at too many inference steps (800), suggesting sensitivity to the intermediate timestep hyperparameter.
Experimental setup
Datasets: MSCOCO Flickr30K
Benchmarks: MSCOCO Flickr30K
Metrics: BLEU METEOR ROUGE-L CIDEr SPICE cosine similarity
Baselines: ZeroCap MAGIC ESPER CLIPRe DeCap CapDec ViECap MeaCap C3
Builds on: DDPM DDIM C3 CapDec CLIP
Closest papers by meaning
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.91)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.90)
- Text-Image Alignment for Diffusion-Based Perception (CVPR 2024 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- ORCA: Hunting Compositional Failures in Text-to-Image Diffusion (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.