Research map / Vision-language and multimodal / Multimodal LLMs
DS-VLM: Diffusion Supervision Vision Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DS-VLM, which adds a training-only Stable Diffusion-based Multi-Adapter Diffusion decoder that reconstructs input images from vision-encoder and connector features, shortening the gradient path from pixels to visual features.
Problem
Gradients from the next-token loss in VLMs must traverse the LLM to reach the vision encoder and connector, causing information loss, and text supervision is semantically sparse relative to images.
Approach
Use a diffusion model conditioned on multi-layer vision-encoder features (low/mid/high) and connector output, via a Mixture-of-Experts cross-attention, to reconstruct the input image with a perceptual reconstruction loss, giving a short gradient path from pixels to visual features; the diffusion model is discarded at inference.
Key results
- With Vicuna-7B and CLIP-L, DS-VLM averages about 1.4% over LLaVA-1.5 across ten benchmarks (e.g., MMBench 61.5 vs 59.1, MMMU 37.8 vs 35.7).
- With Vicuna-13B, DS-VLM reaches MMBench 65.5 vs 64.0 and MME 1836 vs 1781 for LLaVA-1.5.
- Ablation shows perceptual loss outperforms MAE and SSIM reconstruction losses (MMBench 61.5 vs 60.4 for MAE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · MMBench | score: 59.1 -> 61.5 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · MMStar | score: 33.1 -> 34.9 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · MMMU | score: 35.7 -> 37.8 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · MathVista | score: 25.6 -> 27.1 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · OCRBench | score: 31.8 -> 32.4 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · AI2D | score: 55.5 -> 56.7 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · HallusionBench | score: 27.6 -> 28.9 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · LLaVABench | score: 61.8 -> 64.3 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · ScienceQA | score: 69.2 -> 69.7 | Helped |
| Add diffusion supervision to the vision encoder and connector | LLaVA-1.5 with Vicuna-7B · MME | score: 1808 -> 1783 | Hurt |
| Add text and image supervision at high, mid, and low levels (ablation) | DS-VLM with Vicuna-13B · MMBench | score: 64.0 -> 65.5 | Helped |
| Use perceptual loss instead of MAE reconstruction loss (ablation) | DS-VLM with Vicuna-7B · MMBench | score: 60.4 -> 61.5 | Helped |
Surprising findings
- Text supervision alone already improves performance over LLaVA-1.5 in the ablation, with image supervision adding further gains.
Limitations
- Training requires an additional Stable Diffusion-based decoder, increasing training cost (not quantified).
- Gains are modest (about 1-2 points on most benchmarks) and some metrics decline for certain LLM settings (e.g., Llama3-8B MMB 66.3 vs 66.7).
Experimental setup
Datasets: LLaVA-1.5 558K caption pretraining data LLaVA-1.5 665K instruction tuning data Mini-Gemini 1.2M + 1.5M data
Benchmarks: MMBench MM-Star MMMU MathVista OCRBench AI2D HallusionBench LLaVA-Bench ScienceQA MME
Metrics: accuracy score
Baselines: LLaVA-1.5 MiniGPT4 Qwen-VL VisualGLM PandaGPT mPLUG-Owl2 Emu2-chat Yi-VL ShareGPT-4V BLIP-2 Q-Former
Builds on: LLaVA-1.5 Stable Diffusion ControlNet BLIP-2
Closest papers by meaning
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.91)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.88)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.88)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.87)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.87)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.