Research map / Vision-language and multimodal / Multimodal LLMs
VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VILA-U, a unified end-to-end autoregressive model that does image/video understanding and generation with a single next-token objective, using a vision tower trained with contrastive plus reconstruction losses over residual-quantized discrete tokens.
Problem
Unifying visual understanding and generation in one model typically needs external diffusion decoders or yields weak understanding when using VQGAN-style discrete tokens that lack text alignment.
Approach
A unified foundation vision tower (CLIP-initialized, frozen text encoder) trained with text-image contrastive loss plus VQ reconstruction loss using residual quantization, so discrete tokens are both text-aligned and decodable; a single next-token prediction framework with a depth transformer for residual codes handles both tasks.
Key results
- Vision tower reaches 73.3 / 78.0 zero-shot ImageNet top-1 at 256 / 384 resolution, with rFID 1.80 / 1.25.
- On MJHQ-30K, VILA-U (384) achieves FID 7.69 versus 17.77 for LWM and 15.18 for Show-o among autoregressive methods.
- On VQAv2, GQA, POPE, and MME, VILA-U (384) scores 79.4, 60.8, 85.8, and 1401.8, close to continuous-token VLMs such as LLaVA-1.5 and VILA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive loss to reconstruction loss during vision tower training (ablation) | VILA-U vision tower · Visual-language understanding (25M training data) | VQAv2: 57.7 -> 68.0 | Helped |
| add contrastive loss to reconstruction loss during vision tower training (ablation) | VILA-U vision tower · Visual-language understanding (25M training data) | POPE: 75.1 -> 83.7 | Helped |
| increase vision tower training data from 25M to 700M (ablation) | VILA-U vision tower · Visual-language understanding | VQAv2: 68.0 -> 75.3 | Helped |
| increase vision tower training data from 25M to 700M (ablation) | VILA-U vision tower · Visual-language understanding | MM-Vet: 20.8 -> 27.7 | Helped |
| use a contrastive-loss unified vision tower instead of RQ-VAE (ablation) | VILA-U image generation · MJHQ-30K | FID: 12.0 -> 13.2 | Hurt |
| set CFG to 3.0 instead of 1.0 (ablation) | VILA-U image generation · MJHQ-30K | FID: 14.1 -> 12.8 | Helped |
| set CFG to 3.0 instead of 2.0 (ablation) | VILA-U image generation · MJHQ-30K | FID: 13.0 -> 12.8 | Helped |
| set CFG to 5.0 instead of 3.0 (ablation) | VILA-U image generation · MJHQ-30K | FID: 12.8 -> 13.2 | Hurt |
| increase image resolution from 256 to 384 (ablation) | VILA-U · MJHQ-30K | FID: 12.81 -> 7.69 | Helped |
| increase image resolution from 256 to 384 (ablation) | VILA-U · GenAI-Bench basic prompts | Overall VQAScore: 0.76 -> 0.73 | Hurt |
Surprising findings
- Adding the contrastive loss to the vision tower slightly worsens reconstruction (rFID 1.30 to 1.80) and generation FID (12.0 to 13.2) compared with RQ-VAE.
- Training the vector-quantized tower from scratch with both losses yielded only about 5% zero-shot ImageNet top-1 accuracy, motivating the CLIP-initialized recipe.
Limitations
- Contrastive loss degrades reconstruction quality, a trade-off between understanding and generation within one tower.
- Little observed synergy between understanding and generation tasks.
- Generation trained on much less data (15M images) than diffusion baselines, leaving a gap on basic and advanced prompts.
Experimental setup
Datasets: COYO-700M ShareGPT4V MMC4 OpenVid ImageNet MJHQ-30K GenAI-Bench VBench
Benchmarks: VQAv2 GQA TextVQA POPE MME SEED MM-Vet ActivityNet-QA MSVD-QA MSRVTT-QA TGIF-QA MJHQ-30K
Metrics: accuracy FID rFID top-1 accuracy VQAScore VBench total score
Baselines: LLaVA-1.5 VILA Show-o LWM SEED-LLaMA Video-LaVIT Emu2-Chat SD v2.1 SD-XL PixArt Playground v2.5 Open-Sora
Builds on: RQ-VAE CLIP SigLIP VILA VQGAN
Closest papers by meaning
- HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation (NeurIPS 2025 · similarity 0.83)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.83)
- IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting (CVPR 2026 · similarity 0.82)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.82)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.82)
- RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark (CVPR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.