Research map / Image generation / Autoregressive and latent generation
Efficient Generative Modeling with Residual Vector Quantization-Based Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ResGen, a masked discrete-diffusion generator over residual VQ tokens that predicts cumulative RVQ embeddings so sampling steps are independent of RVQ depth, evaluated on ImageNet 256x256 and zero-shot TTS.
Problem
Generative modeling on residual vector quantization (RVQ) token sequences: increasing RVQ depth improves fidelity but typically multiplies the number of sampling steps in autoregressive or masked generators.
Approach
Mask RVQ tokens from the highest depth downward and train the model to predict the summed (cumulative) RVQ embedding of masked tokens with a mixture-of-Gaussians head, rather than individual tokens, so inference steps decouple from both sequence length and depth; the procedure is framed as a discrete diffusion with variational inference.
Key results
- ResGen-rvq16 reaches FID 1.93 with CFG on ImageNet 256x256 at 64 code length and 63 sampling steps, comparable to MAR-L (1.78) and ahead of other baselines of similar size.
- AR-ResGen with 2 refinement steps achieves FID 5.33 (w/ CFG) in 1.56s vs RQ-Transformer's 5.50 in 5.35s, a ~3.4x speed-up.
- On cross-sentence TTS, Rvqvae-ResGen (625M) achieves the lowest WER (1.70) and CER (0.46) among compared models using 25 inference steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict cumulative embeddings instead of discrete tokens directly in parallel across depths (ablation) | ResGen · ImageNet 256×256 | FID (w/ CFG): 2.91 -> 2.43 | Helped |
| increase model size from 576M to 1B parameters (ablation) | ResGen · ImageNet 256×256 | FID (w/ CFG): 2.43 -> 2.26 | Helped |
| replace the depth transformer with an MLP to predict cumulative embeddings (ablation) | RQ-Transformer · ImageNet 256×256 | FID (w/ CFG): 5.50 -> 5.22 | Helped |
| increase AR-ResGen depth-refinement steps from 1 to 8 (ablation) | AR-ResGen · ImageNet 256×256 | FID (w/ CFG): 6.47 -> 5.22 | Helped |
| increase AR-ResGen depth-refinement steps from 1 to 8 (ablation) | AR-ResGen · ImageNet 256×256 | wall-clock time: 1.30s -> 3.05s | Hurt |
| increase RVQ depth from 8 to 16 (ablation) | ResGen · ImageNet 256×256 | FID (w/ CFG): 2.71 -> 1.93 | Helped |
| increase sampling steps (ablation) | ResGen-rvq16 · ImageNet 256×256 | generation quality | Helped |
| use higher top-p values (ablation) | ResGen-rvq16 with CFG · ImageNet 256×256 | generation quality | Helped |
| use lower top-p values (ablation) | ResGen-rvq16 without CFG · ImageNet 256×256 | generation quality | Helped |
| use a moderate sampling temperature (ablation) | ResGen-rvq16 · ImageNet 256×256 | generation quality | Helped |
| train with circle masking and sample with exponential masking instead of circle masking (ablation) | ResGen-rvq16 with CFG · ImageNet 256×256 | FID: 10.09 -> 9.53 | Helped |
| increase MAR-B AR steps from 16 to 64 (ablation) | MAR-B · ImageNet 256×256 | FID: 4.11 -> 2.33 | Helped |
Surprising findings
- Sampling with the exponential masking schedule while training with circle masking gave the best FID (9.53 with CFG), even though the exponential schedule was not used in training.
- A variant predicting discrete tokens directly in parallel across depths performed competitively but underperformed the cumulative-embedding version (FID 12.79 vs 8.77 w/o CFG).
Limitations
- On TTS, the model does not surpass state-of-the-art (Voicebox, DiTTo-TTS) on every metric, and the continuation task is second to DiTTo-en-XL.
- Comparison with MAGVIT-v2 and MaskBit shows lower FID (1.93 vs 1.78 and 1.62), and sampling is disjoint with no revision of previously unmasked tokens.
- Speed comparisons with MAR use different timing protocols (single-image wall-clock vs throughput), which the authors note complicates direct comparison.
Experimental setup
Datasets: ImageNet AI-Hub Korean speech data
Benchmarks: ImageNet 256x256 zero-shot TTS continuation zero-shot TTS cross-sentence
Metrics: FID WER CER SIM-o SIM-r wall-clock time maximum batch size inference steps
Baselines: RQ-Transformer MaskGIT VAR MAR DiT-XL/2 MAGVIT-v2 MaskBit VALL-E SPEAR-TTS CLaM-TTS YourTTS Voicebox
Builds on: MaskGIT RQ-Transformer CLaM-TTS GIVT DiT
Closest papers by meaning
- SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer (CVPR 2025 · similarity 0.89)
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · similarity 0.89)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.88)
- Plug-and-Play Context Feature Reuse for Efficient Masked Generation (NeurIPS 2025 · similarity 0.87)
- JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.87)
- HyperVQ: Enabling Hyperprior Entropy Modeling for VQ-Based Generative Image Compression (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.