Research map / Vision-language and multimodal / Multimodal LLMs
Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPMM-CoT, which uses a Stable Diffusion-style latent diffusion module (VAE + text-conditioned UNet) to learn image features aligned with text for two-stage multimodal chain-of-thought reasoning on ScienceQA, and also applies it to multimodal machine translation.
Problem
Multi-modal chain-of-thought reasoning relies on fixed, shallow image features from off-the-shelf vision models (DETR, CLIP) that are not optimized for reasoning and do not align with language thoughts.
Approach
Replace fixed vision-encoder features with latent-diffusion-learned image features: a VAE encodes the image, noise is added, and a text-conditioned UNet (cross-attention on text representation) predicts the noise, so the image latents are jointly trained with the CoT loss to align with text.
Key results
- DPMM-CoT-Base reaches 90.97% average accuracy on ScienceQA test, +6.06 over MM-CoT-Base (DETR) at 84.91%.
- DPMM-CoT-Large reaches 93.35% average accuracy, +1.67 over MM-CoT-Large (DETR) at 91.68%, and exceeds ChatGPT (73.97%) by 18.18 points.
- DPMM-MT improves over IKD-MMT on Multi30K EN-DE Test17 (36.18 vs 33.83 BLEU) and EN-FR Test16 (66.91 vs 62.53 BLEU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| diffusion-based latent space learning (DPMM-CoT) replacing fixed DETR image features | MM-CoT Base · ScienceQA test | Avg accuracy (%): +6.06 | Helped |
| diffusion-based latent space learning replacing fixed DETR image features | MM-CoT Large · ScienceQA test | Avg accuracy (%): +1.67 | Helped |
| diffusion-based latent space learning | MM-CoT Base · ScienceQA rationale generation | ROUGE-L: +1.21 | Helped |
| diffusion-based latent space learning on mT5-large | mT5 multimodal translation · Multi30K EN-FR Test16 / EN-DE Test17 | BLEU: +4.38 (EN-FR Test16), +2.35 (EN-DE Test17) | Helped |
| removing Stable Diffusion pre-trained initialization (random VAE and UNet) (ablation) | DPMM-CoT Base · ScienceQA test | Avg accuracy (%): -4.88 | Hurt |
| removing UNet (VAE only) (ablation) | DPMM-CoT Base · ScienceQA test | Avg accuracy (%): -1.53 | Hurt |
| freezing VAE to pre-trained Stable Diffusion weights instead of training with CoT (ablation) | DPMM-CoT Base · ScienceQA test | Avg accuracy (%): -1.98 | Hurt |
| zero tensor instead of blank image for image-less questions (ablation) | DPMM-CoT Base · ScienceQA no-image questions | Avg accuracy (%): +1.20 | Helped |
| scale factor tau varied from 0.15215 to 0.20215 (0.18215 default) (ablation) | DPMM-CoT Base · ScienceQA test | Avg accuracy (%) | No clear effect |
| adding image features via diffusion (vs no image features) in mT5 translation | mT5-large · Multi30K EN-DE Test16 and MSCOCO | BLEU: Test16 EN-DE: 41.63 vs Gated Fusion 41.96 (DPMM-MT slightly below) | No clear effect |
Surprising findings
- Using zero tensors instead of blank images for image-less questions gives 1.20 points higher accuracy, which the authors attribute to blank images introducing misleading information in the diffusion process.
- Diffusion components randomly initialized (without Stable Diffusion pre-training) still outperform MM-CoT (DETR), at 86.09% vs 84.91%.
Limitations
- Gains on ScienceQA are measured against baselines whose setup (e.g., image captions excluded) differs and the baseline MM-CoT numbers are partly taken from prior work.
- Ablations and analysis are on a single benchmark; the MMT generalization is limited to Multi30K.
- ROUGE-L may not reflect rationale correctness, as the authors themselves note.
Experimental setup
Datasets: ScienceQA Multi30K Flickr30K MSCOCO test set
Benchmarks: ScienceQA Multi30K (Test16, Test17, MSCOCO)
Metrics: accuracy ROUGE-L BLEU
Baselines: MM-CoT (DETR) MM-CoT (CLIP) UnifiedQA UnifiedQA w/ CoT ChatGPT (GPT-3.5) ChatGPT w/ CoT VisualBERT Patch-TRM mT5 IKD-MMT Gated Fusion RMMT
Builds on: MM-CoT UnifiedQA Stable Diffusion IKD-MMT
Closest papers by meaning
- Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (NeurIPS 2024 · similarity 0.91)
- Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought (CVPR 2026 · similarity 0.90)
- Chain-of-Thought Guided Multi-Modal Object Re-Identification (CVPR 2026 · similarity 0.90)
- Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.90)
- CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models (AAAI 2025 · similarity 0.90)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.