Research map / Vision-language and multimodal / Multimodal LLMs
Auto-Encoding Morph-Tokens for Multimodal LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes morph-tokens, a discrete abstract visual token set whose post-MLLM counterparts are decoupled from pre-MLLM tokens, enabling a unified MLLM for image comprehension and generation via a 3-stage auto-encoding training scheme.
Problem
Unified multimodal LLMs struggle to jointly do visual comprehension and generation because comprehension needs abstracted visual tokens while generation needs visually complete tokens, which conflict when the pre- and post-MLLM tokens must be equal.
Approach
Encode images into discrete abstract morph-tokens via a deconfounded causal Q-former (initialized from a ViT-VQGAN codebook) that serve as visual prompts for comprehension; the MLLM separately generates post-MLLM morph-tokens that are decoded by an auto-regressive decoder and a VQGAN decoder for generation, detaching the text and image reconstruction losses.
Key results
- Achieves 28.0 MMC and 27.4 VST on DEMON, the best among compared MLLMs, and 7.8% higher KGQA than VPG-C.
- Text-to-image CLIP similarity on MS-COCO Karpathy test 70.6 vs 68.6 for Emu2-Gen.
- Image editing L1 on EVR 15.3 vs 22.8 for Emu2-Gen and 16.3 for MGIE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace morph-tokens with complete visual tokens before and after the MLLM (detail-detail) (ablation) | Morph-Token · DEMON MMC | score: 28.0 -> 16.6 | Hurt |
| Use abstract tokens for both pre- and post-MLLM, with SD generation (ablation) | Morph-Token · DEMON MMC | score: 28.0 -> 22.7 | Hurt |
| Use abstract tokens for both pre- and post-MLLM, with a VQ decoder (ablation) | Morph-Token · DEMON MMC | score: 28.0 -> 19.8 | Hurt |
| Remove the additional decoder (ablation) | Morph-Token · DEMON MMC | score: 28.0 -> 18.0 | Hurt |
| Remove deconfounding from the encoder (ablation) | Morph-Token · DEMON MMC | score: 28.0 -> 26.2 | Hurt |
| Replace discrete morph-tokens with continuous visual tokens and use MSE regression (ablation) | Morph-Token · DEMON MMC | score: 28.0 -> 27.0 | Hurt |
Surprising findings
- Continuous morph-token variant with MSE loss performs close to discrete version on comprehension, showing discretization is not critical in some settings.
Limitations
- Generation fidelity limited by reliance on a VQGAN decoder and an auxiliary decoder.
- Ablations show performance drops when morph-tokens are continuous or without deconfounding.
Experimental setup
Datasets: CC3M LAION JourneyDB DiffusionDB IPr2Pr LLaVA SVIT GSD
Benchmarks: NoCaps Flickr30K GQA VSR MME DEMON MS-COCO EVR MA5K MagicBrush
Metrics: CIDEr accuracy CLIP similarity L1 CVS LPIPS Recall@K
Baselines: Emu2-Chat Emu2-Gen SEED-LLaMA LaVIT VPG-C InstructBlip MGIE InsPix2Pix
Builds on: Q-former VQGAN Vicuna SEED-LLaMA TEAL
Closest papers by meaning
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.90)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.88)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.88)
- Making LLaMA SEE and Draw with SEED Tokenizer (ICLR 2024 · similarity 0.87)
- A More Word-like Image Tokenization for MLLMs (CVPR 2026 · similarity 0.87)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.