Research map / Vision-language and multimodal / Multimodal LLMs
I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ThinkDiff, which aligns a VLM to the LLM decoder of a T5-based encoder-decoder (proxy task via vision-language training) so the aligned features transfer to FLUX.1-dev diffusion decoders, yielding multimodal in-context reasoning image generation.
Problem
Text-to-image diffusion models lack multimodal in-context reasoning; existing multimodal diffusion adapters rely on pixel-level reconstruction finetuning that needs scarce reasoning datasets.
Approach
Align a VLM to the LLM decoder of T5 (the same encoder-decoder LLM whose encoder feeds the diffusion model's prompt embedding) using only image-caption data with text cross-entropy; since the decoder and diffusion model share an input feature space, the alignment transfers to the diffusion decoder at inference without reasoning datasets.
Key results
- ThinkDiff-LVLM reaches 0.463 average 4-shot CoBSAT accuracy versus 0.192 for prior SoTA SEED-LLaMA, trained for 5 hours on 4 A100 GPUs.
- 2-shot ThinkDiff-LVLM achieves SoTA on 9 of 10 CoBSAT tasks, exceeding prior best by over 20% on Action-I, Color-II and Action-II.
- Without random masked training the 2-shot average falls well below the full method (e.g., Action-I 0.256 vs 0.459), and using input-token features collapses accuracy to near zero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ThinkDiff-LVLM instead of SEED-LLaMA | CoBSAT 4-shot evaluation · CoBSAT 4-shot average | accuracy: 0.192 -> 0.463 | Helped |
| Add random masked training to prevent shortcut mapping (ablation) | ThinkDiff-LVLM · CoBSAT 2-shot, Action-I | accuracy: 0.256 -> 0.459 | Helped |
| Use generated-token features instead of input-token features (ablation) | ThinkDiff-LVLM · CoBSAT 2-shot, Action-I | accuracy: 0.011 -> 0.459 | Helped |
| Initialize aligner RMSNorm from the LLM encoder's final RMSNorm (ablation) | ThinkDiff-LVLM training · Training convergence | training loss | Helped |
Surprising findings
- Using deep features of input tokens instead of generated tokens collapses CoBSAT accuracy to near zero (e.g., 0.004 to 0.032 per task).
- Baselines SEED-LLaMA, Emu and GILL drop in accuracy from 2-shot to 4-shot, while ThinkDiff-LVLM improves by 4.7%.
Limitations
- Still struggles with certain complex reasoning cases; stronger VLMs, better data and advanced diffusion models may be needed.
- Focuses on logical reasoning rather than image fidelity, limiting use in tasks like image editing.
- Evaluation limited mainly to CoBSAT; more diverse reasoning tasks needed.
Experimental setup
Datasets: CC3M CC12M SBU CoBSAT
Benchmarks: CoBSAT
Metrics: accuracy
Baselines: SEED-LLaMA Emu GILL FLUX1.1-pro-Ultra
Builds on: FLUX.1-dev T5 Qwen2-VL EVA-CLIP CoBSAT
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.92)
- DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models (ICML 2026 · similarity 0.91)
- MMaDA: Multimodal Large Diffusion Language Models (NeurIPS 2025 · similarity 0.91)
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.91)
- Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents (NeurIPS 2025 · similarity 0.90)
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.