Research map / Vision-language and multimodal / Human and embodied modeling
Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Tar, a unified MLLM built on a fully discrete, text-aligned visual tokenizer (TA-Tok) whose codebook is projected from Qwen2.5 LLM embeddings, paired with AR and diffusion generative de-tokenizers.
Problem
Unifying visual understanding and generation in one MLLM is hampered by continuous/pixel representations that break the autoregressive paradigm or conflict between tasks; the paper seeks a shared discrete semantic visual representation.
Approach
A text-aligned codebook is built by projecting frozen LLM token embeddings through a learnable matrix, so visual tokens live in the LLM's embedding space and can be appended to its vocabulary; SigLIP2 features are quantized with scale-adaptive pooling and decoded to SigLIP2 features, and generative AR or diffusion de-tokenizers map the discrete tokens back to pixels.
Key results
- Tar-7B reaches 0.84 GenEval overall (0.85 with Self Reflect) and 84.19 DPG Bench, surpassing listed unified models on GenEval.
- Tar-7B scores 87.8 POPE, 1571 MME-P, 74.4 MMBench and 73.0 SEED, matching Janus-Pro-7B on understanding.
- In controlled comparisons, the TA-Tok codebook used as LLM embedding initialization improves understanding harmonic mean from 91.1 (random) to 92.9 at 25M data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize LLM visual embeddings with the TA-Tok codebook instead of random initialization (ablation) | Tar · MLLM embedding initialization | Und: 91.1 -> 92.9 | Helped |
| initialize LLM visual embeddings with the TA-Tok codebook instead of random initialization (ablation) | Tar · MLLM embedding initialization | Gen: 69.6 -> 70.6 | Helped |
| increase AR-DTok resolution from 256px to 512px (ablation) | AR-DTok 775M · GenEval | GenEval: 0.76 -> 0.79 | Helped |
| increase AR-DTok resolution from 256px to 512px (ablation) | AR-DTok 775M · DPG Bench | DPG: 83.0 -> 82.7 | Hurt |
| reduce AR-DTok size from 775M to 111M (ablation) | AR-DTok · GenEval | GenEval: 0.76 -> 0.75 | Hurt |
| reduce AR-DTok size from 775M to 22M (ablation) | AR-DTok · DPG Bench | DPG: 83.0 -> 81.7 | Hurt |
| use diffusion de-tokenizer instead of AR-DTok 775M at 512px (ablation) | Tar de-tokenizer · GenEval | GenEval: 0.79 vs. 0.77 | No clear effect |
| use 729 rather than 169 image tokens (ablation) | Tar · Understanding tasks | Und: 89.1 -> 92.9 | Helped |
| use fewer image tokens, 169 rather than 729 (ablation) | Tar · Generation at 10M data | DPG Bench: 70.6 -> 72.7 | Helped |
| jointly train understanding and generation with shared TA-Tok representation instead of separate training (ablation) | Tar · Generation | Gen: +5.3 (65.3 -> 70.6) | Helped |
| add I2I and TI2I pretraining tasks (ablation) | Tar · Unified multimodal pretraining | Gen: 66.4 -> 70.7 | Helped |
| add Self Reflect training (ablation) | Tar-7B · GenEval | overall: 0.84 -> 0.85 | Helped |
Surprising findings
- Janus-style separate representations show no gain from joint training for generation, while shared VQVAE and TA-Tok representations gain 8.1% and 5.3% on generation.
- Longer visual token sequences (729) do not improve generation at 10M training data; 81 tokens were best there, and 169 tokens were judged sufficient later.
Limitations
- Vector quantization causes information loss, hurting fine-grained tasks such as OCR.
- Generative de-tokenizers reconstruct input images less faithfully than traditional tokenizers.
- Diffusion de-tokenizer is more computationally expensive at inference.
Experimental setup
Datasets: LAION-5B LLaVA-Recap-CC12M DataComp LLaVA-v1.5 instruction data LLaVA-Next ImageNet-1K ImageNet-21K Megalith-10M JourneyDB Midjourney-Prompts Magpie WebInstruct
Benchmarks: POPE MME MMBench SEED GQA MMMU GenEval DPG Bench ImageNet 256x256 FID
Metrics: accuracy MME-P MME-C GenEval overall DPG overall FID IS harmonic mean of understanding benchmarks
Baselines: Janus Janus-Pro Show-o Harmon ILLUME VILA-U UniTok Chameleon Emu3 LLaVA-v1.5 MetaMorph
Builds on: SigLIP2 LLaMA-based LlamaGen SANA Qwen2.5 UniTok
Closest papers by meaning
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.92)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.91)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.91)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.91)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.