Research map / Vision-language and multimodal / CLIP and zero-shot
Diffusion Feedback Helps CLIP See Better
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIVA, a self-supervised post-training method that fine-tunes CLIP image encoders using reconstruction loss from a frozen text-to-image diffusion model conditioned on CLIP's dense visual features, improving fine-grained visual perception without image-text pairs.
Problem
CLIP models struggle to distinguish fine-grained visual details (orientation, quantity, color, structure), which limits CLIP-based MLLMs and vision models; existing fixes rely on image-text data.
Approach
Use generative feedback from a frozen pre-trained text-to-image diffusion model, conditioned on CLIP's class token plus randomly selected local patch tokens (visual dense recap) and empty text, to backpropagate denoising reconstruction loss into the CLIP vision encoder using only images.
Key results
- On MMVP-VLM, DIVA improves OpenAI ViT-L-14/224 average from 19.3 to 25.9 (+6.6) and MetaCLIP ViT-H-14 from 25.2 to 31.9 (+6.7).
- LLaVA-1.5-7B with DIVA-enhanced CLIP raises MMVP from 24.7 to 31.3 while POPE, MMBench and LLaVA-Wild also improve.
- Zero-shot classification averages on 27 datasets stay essentially unchanged (e.g., OpenAI ViT-L-14 69.3 before and after; MetaCLIP ViT-H-14 74.4 vs 74.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIVA diffusion feedback to enhance OpenAI ViT-L-14 | OpenAI ViT-L-14 · MMVP-VLM (224²) | average accuracy: 19.3 -> 25.9 | Helped |
| Use DIVA-enhanced visual encoder | LLaVA-1.5-7B · MME | score: 1510.7 -> 1500.6 | Hurt |
| Use DIVA-enhanced visual encoder | SAN with ViT-L-14/224 · ADE20K-847 | score: 10.9 -> 11.0 | Helped |
| Use DIVA on OpenAI ViT-L-14 | OpenAI ViT-L-14 · ImageNet-1K zero-shot | top-1 accuracy: 75.5 -> 75.5 | No clear effect |
| Replace empty text condition with real image-matched captions (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 19.3 -> 21.5 | Helped |
| Replace empty text condition with free-source phrases (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 19.3 -> 20.7 | Helped |
| Use only the visual class token instead of partial patch tokens (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 25.9 -> 22.2 | Hurt |
| Use all local patch tokens instead of partial patch tokens (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 25.9 -> 20.0 | Hurt |
| Increase training data to 50% (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 19.3 -> 22.2 | Helped |
| Increase training data to 75% (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 19.3 -> 23.7 | Helped |
| Use DiT-XL/2 as the generative diffusion model (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 19.3 -> 13.3 | Hurt |
| Use SD-1-4 as the generative diffusion model (ablation) | DIVA with OpenAI ViT-L-14/224 · MMVP-VLM | average accuracy: 19.3 -> 23.0 | Helped |
Surprising findings
- Using a DiT-XL/2 diffusion model as guidance decreased MMVP-VLM performance (19.3 to 13.3), attributed to its weaker representation quality versus Stable Diffusion.
- Conditioning on all local patch tokens (too dense a recap) reduced gains (+0.7) compared with partial local tokens (+6.6).
Limitations
- Data scale and model capacity of the generative fine-tuning could be scaled further.
- Only image-text data explored; video and audio modalities are untested.
- Performance gains on MME are absent and some zero-shot retrieval/classification datasets show small drops.
Experimental setup
Datasets: Conceptual-3M
Benchmarks: MMVP-VLM MMVP POPE MME MMBench LLaVA-Bench-in-the-Wild ADE20K-847 ADE20K-150 Pascal Context-459 Pascal Context-59 ImageNet-1K CIFAR-10
Metrics: accuracy recall@K mIoU top-1 accuracy
Baselines: OpenAI CLIP ViT-L-14 MetaCLIP ViT-H-14 SigLIP ViT-SO-14 DFN ViT-H-14 LLaVA-1.5 SAN
Builds on: CLIP Stable Diffusion Diffusion-TTA LLaVA-1.5 SAN
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.92)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.92)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.91)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.91)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.91)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.