Research map / Vision-language and multimodal / CLIP and zero-shot
Visual Lexicon: Rich Image Features in Language Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViLex, an image encoder that maps images into the text vocabulary embedding space of a frozen text-to-image diffusion model, trained self-supervised as an autoencoder, so the resulting tokens serve both as "text" prompts for image generation and as vision features for VLMs.
Problem
Existing image representations either capture high-level semantics (CLIP, SigLIP, DINO) or pixel-level detail (VAE, MAE), and text-based image inversion loses visual detail that language cannot express.
Approach
Use a frozen T2I diffusion model as the decoder in an autoencoder whose encoder (ViT + attention pooling) outputs tokens aligned with the text encoder's vocabulary lookup matrix, with TailDrop to order tokens by information content and text-free guidance to combine ViLex tokens with natural-language prompts.
Key results
- With a single ViLex token, FID on MSCOCO-64x64 is 3.89 versus 3.65 for DeDiffusion's discrete tokens... with 75 tokens ViLex reaches FID 2.07 (Table 1 reports 2.38 for ViLex at 75 tokens vs DeDiffusion's values).
- Human studies show ViLex wins 98% on layout, 95% on semantic and 98% on style against DeDiffusion, and 91%/76%/90% against DALL·E 3 (Table 2).
- Replacing SigLIP with ViLex in PaliGemma-2B raises VQAv2 from 81.4 to 83.6 and SciQA from 85.9 to 88.6 while freezing the encoder, with 25% more tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ViLex tokens instead of DeDiffusion text tokens | Imagen image reconstruction · MSCOCO-64×64 | FID: 3.89 -> 2.07 | Helped |
| Use ViLex tokens instead of DeDiffusion text tokens | Imagen image reconstruction · MSCOCO-64×64 | IS: 14.68 -> 15.88 | Helped |
| Use ViLex for image-to-image generation instead of DeDiffusion | COCO image-to-image · COCO image-to-image | layout preference: 98% | Helped |
| Use ViLex for image-to-image generation instead of DALL·E 3 | COCO image-to-image · COCO image-to-image | semantic preference: 76% | Helped |
| Fine-tune SigLIP with the ViLex training approach | SigLIP vision encoder · Image reconstruction | FID: 2.54 -> 2.38 | Helped |
| Fine-tune SigLIP with the ViLex training approach | SigLIP vision encoder · COCOcap | score: 139.7 -> 141.5 | Helped |
| Fine-tune SigLIP with the ViLex training approach | SigLIP vision encoder · VQAv2-Val | score: 81.4 -> 81.6 | Helped |
| Fine-tune SigLIP with the ViLex training approach | SigLIP vision encoder · RC-val | score: 66.2 -> 67.6 | Helped |
| Add ViLex tokens to the vision-language model | PaliGemma · TextVQA | score: 12.9 -> 59.4 | Helped |
| Add ViLex tokens to the vision-language model | PaliGemma · RC-val | score: 19.5 -> 70.9 | Helped |
| Replace SigLIP with ViLex as the image encoder | PaliGemma-2B · VQAv2 | score: 81.4 -> 83.6 | Helped |
| Replace SigLIP with ViLex as the image encoder | PaliGemma-2B · SciQA | score: 85.9 -> 88.6 | Helped |
Surprising findings
- A single continuous ViLex token already beats the full discrete-token DeDiffusion pipeline on FID and IS.
- Replacing the VAE encoder with ViLex in a VLM is reported to give vision-language scores often more than 8 times higher, indicating pixel-level latents are poor for understanding.
Limitations
- Relies on proprietary Imagen T2I model and WebLI data, limiting reproducibility.
- Reconstruction quality is limited by the 64x64 resolution of the diffusion decoder used in evaluation.
Experimental setup
Datasets: WebLI MSCOCO TextCaps SciCap VQAv2 TextVQA OKVQA ScienceQA VizWiz-QA GQA RefCOCO MSRVTT
Benchmarks: MSCOCO-64x64 FID/IS COCO captioning COCO-35L TextCaps SciCap VQAv2 TextVQA OKVQA ScienceQA VizWiz-QA GQA RefCOCO
Metrics: FID Inception Score human preference win rate CIDEr accuracy
Baselines: DeDiffusion DALL·E 3 DreamBooth Original SigLIP VAE LLaVA1.5-13B VILA-13B QwenVL-7B LLaVA1.6-13B
Builds on: Imagen DeDiffusion SigLIP PaliGemma DreamBooth Classifier-free guidance SoundStream
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.87)
- What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Language Model as Visual Explainer (NeurIPS 2024 · similarity 0.87)
- MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.