AI Paper Atlas

Research map / Image generation / Text-to-image diffusion

Are Diffusion Models Vision-And-Language Reasoners?

Benno Krojer, Elinor Poole-Dayan, Vikram Voleti, Christopher Pal, Siva Reddy

NeurIPS 2023 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Normalize conditional denoising error by subtracting unconditional denoising errorStable Diffusion image retrieval · GDBench image retrievalaccuracy: 11.1 -> 61.5Improved
Fine-tune on MS-COCO without negatives (ablation)DiffusionITM · Flickr30K text retrievalaccuracy: 55.3 -> 62.2Improved
Use CLIP-based hard image negatives (ablation)DiffusionITM · Flickr30K image retrievalaccuracy: 46.1 -> 52.9Improved
Use compositional hard text negatives (ablation)DiffusionITM · Flickr30K text retrievalaccuracy: 55.3 -> 60.9Improved
Use compositional hard text negatives (ablation)DiffusionITM · Flickr30K image retrievalaccuracy: 46.1 -> 47.0Hurt
Use random text negatives (ablation)DiffusionITM · Flickr30K text retrievalaccuracy: 55.3 -> 62.2Improved
Combine hard text, random text, and hard image negatives (ablation)DiffusionITM · Flickr30K image and text retrievalaccuracy: 46.1 -> 49.4; 55.3 -> 61.7Mixed
Remove clipping of the negative loss (ablation)HardNeg-DiffusionITM · ARO COCO Order and Petsaccuracy: 34.9 -> 63.1; 60.9 -> 53.2Mixed
Use image-unconditional denoising as the text-retrieval normalization (ablation)DiffusionITM · Text retrievalaccuracyHurt
Increase the number of noise-timestep samples (ablation)DiffusionITM · Flickr30K text retrievalaccuracyImproved
Fine-tune Stable Diffusion with hard negativesStable Diffusion image generation · DrawBenchhuman preference wins: 60 vs. 33Improved
Use Stable Diffusion 2.1 instead of Stable Diffusion 1.5Stable Diffusion · Bias evaluationaverage absolute effect size: 0.79 -> 0.65Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy