Research map / Vision-language and multimodal / Multimodal LLMs
DEEM: Diffusion models serve as the eyes of large language models for image perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEEM, which uses a diffusion model's generative feedback as a self-supervised consistency regularizer to align a CLIP-ConvNeXt image encoder inside an interleaved LMM, improving robustness and reducing hallucination.
Problem
Large multimodal models relying on CLIP-style image encoders capture biased or task-irrelevant semantics and fail on out-of-distribution images and visual hallucination tests.
Approach
Reuse the LMM's own diffusion image decoder as a generative checker: conditioning the diffusion model on the image encoder's features and adding a reconstruction loss (consistency semantic regularization) corrects the encoder's semantics without extra modules or trainable parameters.
Key results
- DEEM improves over the MM-Interleaved baseline trained under the same setting by about 4% on RobustVQA, 12.8% on POPE and 6.5% on MMVP (average).
- DEEM-VQA reaches 65.56 average accuracy across seven robustness/hallucination/perception benchmarks, versus 53.31 for MM-Interleaved (13B).
- With CLIP-ConvNeXt-B (122M) and Vicuna-7B, DEEM matches or beats DreamLLM (ViT-L) on six of seven shared image-level VQA/captioning splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add consistency semantic regularization during pre-training (ablation) | DEEM · RobustVQA-R | accuracy: 56.1 -> 69.0 | Helped |
| add consistency semantic regularization during pre-training (ablation) | DEEM · POPE-R | accuracy: 50.6 -> 69.9 | Helped |
| add consistency semantic regularization during pre-training (ablation) | DEEM · SPEED | training time: 8.11 s/step -> 9.25 s/step | Hurt |
| add consistency semantic regularization during image-text instruction fine-tuning (ablation) | DEEM · RobustVQA-V | accuracy: 57.4 -> 58.1 | Helped |
| add consistency semantic regularization during image-text instruction fine-tuning (ablation) | DEEM · POPE-R | accuracy: 85.4 -> 86.0 | Helped |
| increase training data from 32K to 96K (ablation) | DEEM, ConvNext-B/Vicuna 7B · POPE-R | accuracy: 48.44 -> 54.42 | Helped |
| increase training data from 96K to 160K (ablation) | DEEM, ConvNext-B/Vicuna 7B · POPE-R | accuracy: 54.42 -> 60.28 | Helped |
| scale image encoder from ConvNext-B to ConvNext-L (ablation) | DEEM, Vicuna 7B, 160K training data · RobustVQA-R | accuracy: 58.93 -> 60.47 | Helped |
| scale language model from Vicuna 7B to Vicuna 13B (ablation) | DEEM, ConvNext-B, 160K training data · OK-VQA | accuracy: 23.65 -> 31.11 | Helped |
| add consistency semantic regularization (ablation) | LLaVA · MMVP | accuracy: 18.6 -> 25.1 | Helped |
| add consistency semantic regularization (ablation) | DEEM · MMVP | accuracy: 20.3 -> 28.7 | Helped |
| increase image resolution from 256 to 448 during instruction fine-tuning (ablation) | DEEM · OK-VQA | accuracy: 38.7 -> 41.0 | Helped |
Surprising findings
- Without SFT, the pre-trained DEEM scores near chance on POPE (around 50%), indicating the pre-trained model answers yes/no poorly before instruction tuning.
- Instruction fine-tuning degrades robustness on RobustVQA, which the consistency regularization partially mitigates.
Limitations
- Robustness knowledge is only partially retained after supervised fine-tuning (forgetting is alleviated, not eliminated).
- Relies on a CLIP-ViT-based teacher for hard-example mining and still depends on a separate image encoder, which raises cost when scaled to larger encoders.
Experimental setup
Datasets: MMC4-Core LAION-400M SBU CC-12M LLaVA-665K COCO Caption VQAv2 TextCaps OCR-VQA GQA OK-VQA TextVQA
Benchmarks: RobustVQA POPE MMVP COCO Image2Paragraph VQAv2 OK-VQA GQA VizWiz VisDial MMBench MMVet
Metrics: accuracy CIDEr METEOR FID NDCG
Baselines: MM-Interleaved DreamLLM SEED-X SEED Emu-I Shikra NeXT-Chat LLaVA-1.5 Qwen-VL
Builds on: MM-Interleaved LLaVA Stable Diffusion v2.1 Perceiver Resampler CLIP
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
- What Matters When Repurposing Diffusion Models for General Dense Perception Tasks? (ICLR 2025 · similarity 0.91)
- DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks (NeurIPS 2025 · similarity 0.90)
- Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models (ICLR 2025 · similarity 0.90)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.