Research map / Image generation / Autoregressive and latent generation
MergeVQ: A Unified Framework for Visual Generation and Representation with Disentangled Token Merging and Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MergeVQ, a VQ tokenizer using ToMe token merging plus LFQ with a learned source-matrix recovery, and MergeAR (KV-cache pruning of duplicate tokens) for unified ImageNet representation learning and generation.
Problem
Visual generation and representation learning compete in a shared VQ latent space, causing a trade-off in quality, representation accuracy, and efficiency.
Approach
Use token merging (ToMe) in the encoder to compress latent space into K semantic tokens while storing a source matrix of spatial positions; quantize with LFQ, recover all tokens via the source matrix for reconstruction, and align the merged tokens globally with DINOv2. For generation, MergeAR prunes duplicate tokens in the KV cache, and the source matrix can also drive RandAR-style random-order ge
Key results
- MergeVQ (R) with 36 tokens reaches 79.8% linear probing and 84.2% fine-tuning top-1 on ImageNet-1K, versus DINOv2 (196 tokens) at 84.5% linear probing.
- MergeVQ (G) with MergeAR achieves gFID 3.05 and IS 260.9 without CFG on ImageNet 256x256 class-conditional generation, and gFID 2.24 with CFG and RandAR.
- MergeVQ (G+R) with 144 tokens reaches rFID 1.48, compared to 1.16 for MAGVIT-v2 with 256 tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MergeVQ (R) instead of DINOv2 for self-supervised pre-training | ImageNet-1K linear probing · ImageNet-1K | top-1 accuracy: 84.5 -> 79.8 | Hurt |
| Use MergeVQ (R) instead of DINOv2 for self-supervised pre-training | ImageNet-1K fine-tuning · ImageNet-1K | top-1 accuracy: 85.7 -> 84.2 | Hurt |
| Use MergeVQ (G+R) rather than MergeVQ (R), jointly training for generation and representation | MergeVQ · ImageNet-1K linear probing | top-1 accuracy: 79.8 -> 77.9 | Hurt |
| Increase MergeVQ (G) kept tokens from 64 to 256 (ablation) | MergeVQ (G) tokenizer · ImageNet-1K reconstruction | rFID: 4.74 -> 1.41 | Helped |
| Increase MergeVQ (G+R) kept tokens from 36 to 144 (ablation) | MergeVQ (G+R) tokenizer · ImageNet-1K reconstruction | rFID: 8.94 -> 3.07 | Helped |
| Add Source Recovery using 2-layer Cross-Attention instead of ground-truth source matrix (ablation) | MergeVQ (G+R) · ImageNet-1K reconstruction | rFID: 1.48 -> 1.71 | Hurt |
| Use MergeAR KV cache compression rather than vanilla LlamaGen (ablation) | MergeVQ (G+R) · ImageNet-1K class-conditional generation | gFID: 3.28 -> 3.25 | Helped |
| Use MergeAR KV cache compression rather than vanilla LlamaGen (ablation) | MergeVQ (G) · ImageNet-1K class-conditional generation | gFID: 3.14 -> 3.05 | Helped |
Surprising findings
- Representation learning benefits from fewer merged tokens (36) while reconstruction benefits from more (256), so the same latent space shows a clear trade-off between the two tasks.
- MergeAR's KV-cache pruning gives only a small gFID improvement over vanilla LlamaGen (0.09 vs 0.03), suggesting limited benefit unless the sequence is redundant.
Limitations
- MergeAR gain over vanilla LlamaGen is marginal in gFID.
- Representation and generation remain in tension; G+R version trails the R-only version on linear probing.
- Some results (e.g., RandAR combination) are described as to be updated in the arXiv version.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: linear probing top-1 accuracy fine-tuning top-1 accuracy rFID gFID Inception Score
Baselines: MAGVIT-v2 OpenMAGVIT2 LlamaGen RandAR DiGIT MAGE-C DINOv2 MaskGIT VAR MAR
Builds on: VQGAN ToMe LlamaGen RandAR DINO MAGVIT-v2
Closest papers by meaning
- SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer (CVPR 2025 · similarity 0.87)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.87)
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · similarity 0.87)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.87)
- CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs (CVPR 2026 · similarity 0.87)
- Channel-wise Vector Quantization (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.