Research map / Image generation / Text-to-image diffusion
Are Diffusion Models Vision-And-Language Reasoners?
Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, Siva Reddy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Normalize conditional denoising error by subtracting unconditional denoising error | Stable Diffusion image retrieval · GDBench image retrieval | accuracy: 11.1 -> 61.5 | Improved |
| Fine-tune on MS-COCO without negatives (ablation) | DiffusionITM · Flickr30K text retrieval | accuracy: 55.3 -> 62.2 | Improved |
| Use CLIP-based hard image negatives (ablation) | DiffusionITM · Flickr30K image retrieval | accuracy: 46.1 -> 52.9 | Improved |
| Use compositional hard text negatives (ablation) | DiffusionITM · Flickr30K text retrieval | accuracy: 55.3 -> 60.9 | Improved |
| Use compositional hard text negatives (ablation) | DiffusionITM · Flickr30K image retrieval | accuracy: 46.1 -> 47.0 | Hurt |
| Use random text negatives (ablation) | DiffusionITM · Flickr30K text retrieval | accuracy: 55.3 -> 62.2 | Improved |
| Combine hard text, random text, and hard image negatives (ablation) | DiffusionITM · Flickr30K image and text retrieval | accuracy: 46.1 -> 49.4; 55.3 -> 61.7 | Mixed |
| Remove clipping of the negative loss (ablation) | HardNeg-DiffusionITM · ARO COCO Order and Pets | accuracy: 34.9 -> 63.1; 60.9 -> 53.2 | Mixed |
| Use image-unconditional denoising as the text-retrieval normalization (ablation) | DiffusionITM · Text retrieval | accuracy | Hurt |
| Increase the number of noise-timestep samples (ablation) | DiffusionITM · Flickr30K text retrieval | accuracy | Improved |
| Fine-tune Stable Diffusion with hard negatives | Stable Diffusion image generation · DrawBench | human preference wins: 60 vs. 33 | Improved |
| Use Stable Diffusion 2.1 instead of Stable Diffusion 1.5 | Stable Diffusion · Bias evaluation | average absolute effect size: 0.79 -> 0.65 | Improved |
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
- Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.91)
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models (CVPR 2024 · similarity 0.90)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.90)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
