Research map / Image generation / Autoregressive and latent generation

Channel-wise Vector Quantization

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Hierarchical VQ-VAE generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Channel-wise Vector Quantization (CVQ), which quantizes each channel of a latent map instead of each spatial patch, and a channel-wise autoregressive model (CAR) that generates images via next-channel prediction, reporting 100% codebook usage and competitive text-to-image GenEval/DPG scores.

Problem

Patch-wise vector quantization suffers from low codebook utilization (codebook collapse) and imposes a 2D grid tokenization that is mismatched to unidirectional next-token autoregressive generation.

Approach

Replace patch-wise quantization with channel-wise quantization so each codebook index encodes a whole h×w channel map; channels are then ordered coarse-to-fine via nested channel dropout, yielding a 1D token sequence for next-channel AR prediction.

Key results

• CVQ achieves 100% codebook usage at 16,384 codebook size with rFID 2.60 and PSNR 20.94 at 256 tokens, versus 4.5% usage and rFID 4.84 for vanilla VQ.
• CAR (8B) reaches GenEval 0.79 and DPG 86.72 among unidirectional models, competitive with Infinity and InfinityStar.
• CAR attains MJHQ-30K FID of 6.42, outperforming the standard 2D-token baseline and 1D masked-token baselines reported in the paper.

Surprising findings

• Nested channel dropout hurts patch-wise VQ reconstruction badly (rFID 15.32) while CVQ keeps reconstruction quality under the same dropout.

Limitations

• Fixed token count limits flexibility; at higher resolution with fixed token budget CVQ trades reconstruction fidelity for efficiency.
• Ordering of channels relies on heuristic nested dropout whose theoretical guarantees apply only to semi-linear autoencoders.
• Generation results for the best models rely on a large in-house and web-scale 80M pair training mix that is not released.

Experimental setup

Datasets: ImageNet-1K ImageNet-21K (filtered) LAION-Aesthetics-12M CC12M Megalith-10M BLIP-3o-short BLIP-3o-long 6M in-house aesthetics dataset

Benchmarks: ImageNet-1K GenEval DPG-Bench MJHQ-30K

Metrics: rFID SSIM PSNR codebook usage GenEval overall DPG overall FID

Baselines: VQGAN SimVQ VQ-LC MoVQ TiTok TA-TiTok FlexTok SpectralAR Emu3 NextStep-1 Infinity InfinityStar

Builds on: VQ-VAE VQGAN nested dropout Qwen3 VAR

Closest papers by meaning

• Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.90)
• TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.89)
• Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.89)
• Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.88)
• UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generation (ICLR 2026 · similarity 0.88)
• UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.87)