Research map / Vision-language and multimodal / Multimodal LLMs
Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a frozen text-to-image diffusion decoder (Stable Diffusion 2.1) into continuous caption embeddings from image-only data, then fine-tunes a Qwen-2.5 LLM to decode these embeddings into detailed captions that rival GPT-4o and Gemini 2.0 Flash.
Problem
Training strong captioning VLMs requires billions of high-quality image-text pairs and millions of GPU hours; the paper asks whether a text-to-image diffusion model's knowledge can be distilled into a language-semantic bottleneck using only single-modal images.
Approach
A vision encoder with learnable queries produces caption embeddings in CLIP text space that condition a frozen T2I diffusion U-Net trained with the denoising loss (no paired text), and a second stage maps these embeddings through frozen CLIP text encoder and an MLP into a Qwen-2.5 LLM that generates captions.
Key results
- Captions from VLV achieve FID 11.47 at guidance 1.0 vs GPT-4o 12.16 and Qwen2.5-VL-7B 12.61 (Table on SD3.5 reconstruction), with best open-source result at guidance 2.0 (6.64).
- In the Captioner Arena with Gemini judge, VLV scores 5.18 average vs GPT-4o 5.23 and Qwen2.5-7B 5.03.
- On text-only VQA at 32-shot, VLV reaches 63.60 on VQAv2 and 60.25 on OK-VQA, vs best 64.05 and 62.31 for Gemini 2.0 Flash.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLV captions instead of Florence-2 Large captions | Caption-based image reconstruction with Stable Diffusion 3.5 Medium · MS-COCO 2014 val, guidance scale 2.0 | FID: 7.51 -> 6.64 | Helped |
| Use VLV captions instead of Gemini 2.0 Flash captions | Caption-based image reconstruction with Stable Diffusion 3.5 Medium · MS-COCO 2014 val, guidance scale 1.0 | FID: 12.82 -> 11.47 | Mixed |
| Use VLV captions instead of GPT-4o captions | Caption-based image reconstruction with Stable Diffusion 3.5 Medium · MS-COCO 2014 val, guidance scales 1.0–4.0 | FID: 12.16 -> 11.47; 6.20 -> 6.64; 7.96 -> 8.56; 9.25 -> 9.90 | Mixed |
| Use VLV captions instead of Qwen-2.5-VL-7B captions | Text-only VQA using DeepSeek-V3 · VQAv2, 0-shot | accuracy (%): 61.74 -> 58.55 | Hurt |
| Provide 32 in-context examples instead of zero | VLV captions used as context for DeepSeek-V3 · OK-VQA | accuracy (%): 45.31 -> 60.25 | Helped |
| Use 77 learnable queries instead of 16 (ablation) | VLV representation learning · MS-COCO 2017 test | FID: 5.72 -> 5.30 | Helped |
| Unfreeze the LLM decoder in addition to training the MLP (ablation) | VLV caption decoder · MS-COCO, guidance scale 2.0 | FID: 9.71 -> 7.55 | Helped |
| Unfreeze the VLV encoder in addition to the MLP and LLM decoder (ablation) | VLV caption decoder · MS-COCO, guidance scale 2.0 | FID: 7.55 -> 6.64 | Helped |
| Train on 40M instead of 6M images (ablation) | VLV autoencoder · MS-COCO 2014 val, guidance scale 2.0 | FID: 9.01 -> 7.22 | Helped |
| Use a Qwen-2.5 3B instead of 0.5B caption decoder (ablation) | VLV caption decoder · MS-COCO 2014 val, guidance scale 2.0 | FID: 9.37 -> 6.64 | Helped |
| Add Stage-1 VLV autoencoding (ablation) | Qwen-2.5 3B caption decoder · MS-COCO 2014, guidance scale 2.0 | FID: 51.03 -> 7.55 | Helped |
Surprising findings
- Skipping stage-1 autoencoding yields very poor reconstruction FID (45 to 68), so the LLM decoder alone cannot recover fidelity without the distilled embeddings.
- VLV is weak on OCR tasks because aesthetic filtering removed text-heavy images.
Limitations
- Poor OCR performance due to aesthetic and watermark filtering of training data
- Built on outdated Stable Diffusion 2.1 decoder, limiting transferable knowledge
- Captions depend on Gemini 2.0 Flash labels for stage-2 alignment
Experimental setup
Datasets: LAION-2B-en-aesthetic (40M subset) MS-COCO 2014 validation MS-COCO 2017 test VQAv2 OK-VQA
Benchmarks: MS-COCO 2014 validation (FID reconstruction) VQAv2 OK-VQA
Metrics: FID rating score (0-6) accuracy (exact match)
Baselines: Florence-2 Large Qwen-2.5-VL-7B Gemini 2.0 Flash GPT-4o LLaVA-v1.5-7B Recap-7B Qwen2.5-7B
Builds on: De-Diffusion ViLex Stable Diffusion 2.1 Florence-2 Qwen-2.5
Closest papers by meaning
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- DS-VLM: Diffusion Supervision Vision Language Model (ICML 2025 · similarity 0.91)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.90)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.