Research map / Vision-language and multimodal / Human and embodied modeling
Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaVIT, a multimodal LLM that tokenizes images into discrete, dynamic-length visual tokens via a selector-merger tokenizer so that images and text share one next-token prediction objective.
Problem
Adapter-based multimodal LLMs treat visual input merely as a prompt with no supervision, and fixed-length visual tokens are computationally costly, limiting unified understanding and generation.
Approach
A dynamic visual tokenizer with a Gumbel-Softmax token selector that keeps informative patches and a token merger that compresses discarded patches into retained ones, then quantizes them into a learnable codebook so images become discrete "foreign language" tokens trained with the same cross-entropy next-token objective as text.
Key results
- Zero-shot LaVIT-7B reaches 83.0 CIDEr on Flickr30k vs 61.5 for Flamingo-9B and 74.9 for BLIP-2 (Vicuna-7B), and 54.6% on OKVQA vs 44.7% for Flamingo-9B.
- Zero-shot text-to-image FID of 7.4 on MS-COCO-30K, better than Emu (LLaMA-13B, 11.7) and CM3Leon-7B (10.8), and comparable to Parti-20B (7.2).
- Dynamic tokenization uses about 94 visual tokens on average (about 36% of 256) and cuts pre-training time from 30h to 18h while improving Flickr, VQAv2 and OKVQA in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace visual token classification with next-visual-feature regression (ablation) | LaVIT · Flickr30k | CIDEr: 73.2 -> 60.4 | Hurt |
| use dynamic rather than fixed visual token length (ablation) | LaVIT · Flickr30k | CIDEr: 71.1 -> 74.0 | Helped |
| use dynamic rather than fixed visual token length (ablation) | LaVIT · Pre-training | training time: 30h -> 18h | Helped |
| replace causal attention with bi-directional attention in the token merger (ablation) | LaVIT · Flickr30k | CIDEr: 73.2 -> 69.9 | Hurt |
| use continuous token-merger features instead of quantized embeddings as LLM visual input (ablation) | LaVIT · Flickr30k | CIDEr: 70.3 -> 72.7 | Helped |
| unlock the LLM during pre-training rather than keeping it frozen (ablation) | LaVIT · Flickr30k | CIDEr: 63.0 -> 72.1 | Helped |
| add the token merger to compress discarded patch semantics onto retained patches (ablation) | LaVIT · Flickr30k | CIDEr: 70.8 -> 72.7 | Helped |
| use LLM learning rate 1.5e-4 rather than 5e-5 during multimodal pre-training (ablation) | LaVIT-7B · MMLU (5-shot) | Average: $33.6 (-1.5)$ -> $27.1 (-8.0)$ | Hurt |
| use 50% text and 50% multimodal data rather than 66% text and 33% multimodal data (ablation) | LaVIT-7B · MMLU (5-shot) | Average: $33.6 (-1.5)$ -> $30.9 (-4.2)$ | Hurt |
| use LaVIT's unified visual token modeling and dynamic tokenizer instead of the strongest reported baseline | LaVIT (LLaMA-7B) · Flickr30k | CIDEr: 80.5 -> 83.0 | Helped |
| use LaVIT's unified visual token modeling and dynamic tokenizer instead of the strongest reported baseline | LaVIT (LLaMA-7B) · OKVQA | accuracy: 44.7 -> 54.6 | Helped |
| use LaVIT for text-to-image generation instead of Parti-20B | LaVIT (LLaMA-7B) · MS-COCO-30K | FID: 7.2 -> 7.4 | Hurt |
Surprising findings
- Replacing the discrete classification objective for visual tokens with feature regression (as in Emu) severely degrades performance on Flickr, VQAv2 and OKVQA in the ablation.
- Unlocking the LLM during pre-training drops MMLU by only 1.5 points at the chosen learning rate, though a higher LLM learning rate causes an 8.0-point drop.
Limitations
- Inherits LLM hallucination and factual error issues.
- Trained only on web-scale image-text pairs with noisy, short captions, limiting modeling of extensive text-image correspondence.
- Tokenizer uses 224x224 inputs and the main model uses frozen-free LLM training that is costly (256 A100 GPUs).
Experimental setup
Datasets: LAION-400M Conceptual Captions SBU BLIP-Capfilt LAION-Aesthetics RedPajama MS-COCO MMLU
Benchmarks: NoCaps Flickr30k VQAv2 OKVQA GQA VizWiz MS-COCO FID-30K MMLU
Metrics: CIDEr VQA accuracy FID MMLU 5-shot accuracy
Baselines: Flamingo-3B Flamingo-9B OpenFlamingo-9B Kosmos-1 Kosmos-2 BLIP-2 CM3Leon-7B Emu GILL Parti Stable Diffusion DALL-E
Builds on: LLaMA EVA-CLIP Stable Diffusion VQ-VAE Gumbel-Softmax
Closest papers by meaning
- Beyond Text: Frozen Large Language Models in Visual Signal Comprehension (CVPR 2024 · similarity 0.92)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.91)
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.91)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.91)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.