Research map / Vision-language and multimodal / CLIP and zero-shot
De-Diffusion Makes Text a Strong Cross-Modal Interface
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes De-Diffusion, an autoencoder that encodes images into interpretable text using a frozen pre-trained text-to-image diffusion model (Imagen-based) as decoder, and uses that text as a cross-modal interface for text-to-image prompting and few-shot multimodal tasks with off-the-shelf LLMs.
Problem
Deep embeddings are the usual cross-modal interface between images and language, but they are not interpretable and often require adaptive training; image captions are interpretable but not content-preserving or comprehensive.
Approach
Train an image-to-text encoder with no text supervision, reconstructing images through a frozen pre-trained text-to-image diffusion model whose CLIP text vocabulary serves as the discrete latent (Gumbel-softmax), so the text becomes a precise, comprehensive, transferable image description.
Key results
- De-Diffusion text achieves FID 6.43 with Stable Diffusion v2-base on MS-COCO, beating human COCO captions (8.08), BLIP-2 and PaLI-X.
- With PaLM 2-L, De-Diffusion reaches 68.4 VQAv2 test-dev and 60.6 OKVQA val at 32 shots, new state of the art over Flamingo-80B (67.6 / 57.8) on these settings.
- Open-ended 5-way 1-shot miniImageNet with PaLM 2-L and task induction reaches 97.0%, above prior closed-form best of 95.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use De-Diffusion text instead of human-annotated COCO captions as Stable Diffusion prompts | Text-to-image reconstruction on MS-COCO 2014 validation · MS-COCO 2014 validation, guidance scale 3.0 | FID: 8.08 -> 6.43 | Helped |
| Add De-Diffusion image context to PaLM 2-L few-shot VQA | De-Diffusion ViT-L + PaLM 2-L · VQAv2 test-dev, 32-shot | accuracy: 67.6 vs 68.4 | Helped |
| Add De-Diffusion image context to PaLM 2-L few-shot VQA | De-Diffusion ViT-L + PaLM 2-L · OKVQA val, 32-shot | accuracy: 57.8 vs 60.6 | Helped |
| Increase PaLM 2-L examples from 0-shot to 32-shot | De-Diffusion ViT-L + PaLM 2-L · MS-COCO caption | CIDEr: 88.5 -> 103.7 | Helped |
| Increase PaLM 2-S examples from 4-shot to 32-shot | De-Diffusion ViT-L + PaLM 2-S · VQAv2 test-dev | accuracy: 64.0 -> 63.1 | Hurt |
| Increase PaLM 2-S examples from 4-shot to 32-shot | De-Diffusion ViT-L + PaLM 2-S · OKVQA val | accuracy: 53.5 -> 53.3 | Hurt |
| Use De-Diffusion image descriptions instead of BLIP-2 captions | PaLM 2-L VQA using image context · VQAv2 val, 0-shot | accuracy: 63.1 -> 65.2 | Helped |
| Add task induction instructions and expected class names | De-Diffusion + PaLM 2-L · miniImageNet, open-ended 5-way 1-shot | accuracy: 71.8 -> 97.0 | Helped |
| Use fewer text tokens (5 instead of 75) (ablation) | De-Diffusion · MS-COCO 2014 validation, Stable Diffusion v2-base | FID: 6.43 -> 9.19 | Hurt |
| Use fewer text tokens (5 instead of 75) (ablation) | De-Diffusion · miniImageNet, 5-way 1-shot with task induction | accuracy: 97.0 -> 97.8 | Helped |
| Exclude punctuation from the token vocabulary (ablation) | De-Diffusion · MS-COCO 2014 validation, Stable Diffusion v2-base | FID: 6.85 -> 6.43 | Helped |
| Include punctuation in the token vocabulary (ablation) | De-Diffusion · miniImageNet, 5-way 1-shot with task induction | accuracy: 97.0 -> 96.8 | No clear effect |
Surprising findings
- Few-shot classification accuracy slightly decreases with more text tokens (97.8% at 5 tokens to 97.0% at 75), while reconstruction FID improves.
- Training the encoder from scratch with only the diffusion reconstruction objective yields 72.2% miniImageNet accuracy, surpassing SPAE's 67.0% with PaLM 2-L.
Limitations
- Text is a discrete latent whose tokens are constrained to the CLIP vocabulary minus punctuation, so readability is imperfect
- Few-shot gains depend on LLM size; smaller PaLM 2-S degrades with long 32-shot prompts
- Transferability evaluated mainly on FID and few-shot tasks rather than broad generation quality
Experimental setup
Datasets: WebLI MS-COCO 2014 miniImageNet VQAv2 OKVQA
Benchmarks: MS-COCO 2014 val reconstruction FID VQAv2 OKVQA MS-COCO captioning miniImageNet 5-way 1-shot
Metrics: FID VQA accuracy CIDEr classification accuracy
Baselines: BLIP-2 PaLI-X Human COCO captions Flamingo-80B IDEFICS-80B SPAE LQAE Frozen P>M>F PICa-Full AnyMAL LENS
Builds on: Imagen CoCa Neural De-Rendering Flamingo SPAE
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.91)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.91)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.90)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.