Research map / Vision-language and multimodal / Human and embodied modeling
End-to-End Vision Tokenizer Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ETT, which makes a VQ vision tokenizer (IBQ-based) differentiable inside an LLM by feeding codebook embeddings instead of discrete indices, and tunes tokenizer and LLM jointly with caption and reconstruction losses.
Problem
Vision tokenizers are optimized for low-level reconstruction and frozen during downstream autoregressive multimodal training, creating a representation bottleneck for understanding and generation tasks.
Approach
Connect the tokenizer's codebook embeddings (via an MLP projector) directly to the LLM so gradients flow into the tokenizer, and jointly optimize caption cross-entropy with the VQ reconstruction loss, without changing the LLM's text vocabulary or heads.
Key results
- ETT with 1.5B LLM achieves GenEval overall 0.63, outperforming SDXL (0.55) and Show-o (0.53) under the stated inference setting.
- Tuning the tokenizer with ETT improves SEED-Bench-Img, GQA, TextVQA and MME-P over frozen codebook embeddings (e.g., TQA 40.9 to 46.9 with 0.25 reconstruction weight), with rFID 1.648 versus 1.033 untuned.
- Understanding-only tuning gives the best understanding scores but rFID degrades to 45.701, showing a trade-off with reconstruction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace discrete indices with codebook embeddings without end-to-end tuning (ablation) | ETT · GQA | accuracy: 50.9 -> 52.6 | Helped |
| replace discrete indices with codebook embeddings without end-to-end tuning (ablation) | ETT · GenEval | overall: 0.40 -> 0.34 | Hurt |
| add end-to-end tokenizer tuning to codebook embeddings (ablation) | ETT · SEEDBench-Img | score: 54.8 -> 60.0 | Helped |
| add end-to-end tokenizer tuning to codebook embeddings (ablation) | ETT · GenEval | overall: 0.34 -> 0.43 | Helped |
| tune the tokenizer for understanding only, without reconstruction loss (ablation) | ETT · GQA | accuracy: 52.6 -> 55.2 | Helped |
| tune the tokenizer for understanding only, without reconstruction loss (ablation) | ETT · ImageNet 256×256 | rFID: 1.033 -> 45.701 | Hurt |
| add reconstruction loss with α=0.25 during understanding tuning (ablation) | ETT · GQA | accuracy: 55.2 -> 54.8 | Hurt |
| add reconstruction loss with α=0.25 during understanding tuning (ablation) | ETT · ImageNet 256×256 | rFID: 45.701 -> 1.648 | Helped |
| increase reconstruction-loss weight α from 0.25 to 1.0 (ablation) | ETT · ImageNet 256×256 | rFID: 1.648 -> 1.500 | Helped |
| increase reconstruction-loss weight α from 0.25 to 1.0 (ablation) | ETT · GQA | accuracy: 54.8 -> 53.9 | Hurt |
Surprising findings
- Understanding-only tuning of the tokenizer collapses reconstruction (rFID 1.033 to 45.701), and adding a small reconstruction weight recovers it only partly.
Limitations
- Relies on fine-tuning existing tokenizers rather than training a tokenizer from scratch for understanding and generation.
- Limited data scale and model capacity for end-to-end fine-tuning.
- Understanding-only tuning severely degrades reconstruction; trade-off requires weight tuning.
Experimental setup
Datasets: SOL-recap SA-1B OpenImages LAION LLaVA-OneVision Infinity-MM LLaVA-mix-665K ImageNet
Benchmarks: GQA TextVQA POPE MME MMBench SEED-Bench MM-Vet ScienceQA-IMG GenEval T2I-CompBench ImageNet rFID
Metrics: accuracy overall score rFID
Baselines: Emu3 Show-o Janus Liquid Chameleon LWM VILA-U LlamaGen SDXL QwenVL-Chat EVE Cambrian
Builds on: IBQ Emu3 VQGAN LPIPS
Closest papers by meaning
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.90)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.89)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.89)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.89)
- UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.