Research map / Image generation / Autoregressive and latent generation
Concept-Centric Token Interpretation for Vector-Quantized Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CORTEX, which trains an information extractor mapping VQGAN token embeddings to ImageNet concepts, then uses saliency-based sample-level and Gumbel-Softmax codebook-level optimization to identify concept-specific tokens for interpretation, masking tests, bias detection, and image editing.
Problem
Vector-quantized generative models have a codebook of discrete tokens whose role in representing concepts is poorly understood; it is unclear which tokens are critical for generating a given concept, and frequency-based token selection picks up background content.
Approach
Train an information extractor that maps token-based embeddings back to concept labels, then derive token importance from its gradients (sample-level) and optimize a Gumbel-Softmax token selection matrix over the full codebook (codebook-level), with the Information Bottleneck motivating filtering of non-essential tokens.
Key results
- Information extractors reach 53.07% Top-1 and 77.73% Top-5 accuracy (CNN-based) on 1000 ImageNet classes from VQGAN embeddings.
- Masking CORTEX concept tokens drops ResNet50 accuracy by 46.6-46.8% versus 41.9% for the frequency baseline while masking fewer tokens (42 vs 64 on average).
- Bias analysis with neutral 'doctor' prompt: white-associated tokens appear 8.55 vs 2.32 per image for black (Cliff's delta 0.456); male 8.27 vs female 3.43 at n=10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ResNet-based IEM-selected concept tokens instead of the frequency-based token baseline | CORTEX sample-level explanation · ImageNet concepts, ResNet50 | ΔP after masking: -0.365 -> -0.404 | Helped |
| Use ResNet-based IEM-selected concept tokens instead of the frequency-based token baseline | CORTEX sample-level explanation · ImageNet concepts, ResNet50 | ΔA after masking: -41.9% -> -46.8% | Helped |
| Optimize codebook token selection end-to-end instead of optimizing embeddings and quantizing afterward | CORTEX codebook-level explanation · Bird category transfer, ResNet18 | ΔP original / ΔP target: -0.224 / 0.127 -> -0.290 / 0.162 | Helped |
| Optimize codebook token selection end-to-end instead of optimizing embeddings and quantizing afterward | CORTEX codebook-level explanation · Bird category transfer, ResNet50 | ΔP original / ΔP target: -0.217 / -0.122 -> -0.282 / 0.165 | Helped |
| Optimize codebook token selection end-to-end instead of optimizing embeddings and quantizing afterward | CORTEX codebook-level explanation · Bird category transfer, ViT-B/16 | ΔP original / ΔP target: -0.174 / 0.072 -> -0.243 / 0.121 | Helped |
| Optimize codebook token selection end-to-end instead of optimizing embeddings and quantizing afterward | CORTEX codebook-level explanation · Bird category transfer, ViT-B/32 | ΔP original / ΔP target: -0.190 / 0.080 -> -0.245 / 0.120 | Helped |
| Analyze neutral-prompt images using white-associated rather than black-associated concept tokens | CORTEX bias analysis on DALL·E-mini · Neutral prompt, doctor images, Top-20 tokens | Mean token frequency per image: 2.32 -> 8.55 | Mixed |
| Analyze neutral-prompt images using male-associated rather than female-associated concept tokens | CORTEX bias analysis on DALL·E-mini · Neutral prompt, doctor images, Top-10 tokens | Mean token frequency per image: 3.43 -> 8.27 | Mixed |
Surprising findings
- The Transformer-based extractor performs worse than CNN and ResNet extractors, which the authors attribute to deliberately shallow design for efficiency.
- Embedding optimization followed by nearest-codebook quantization yields weaker and sometimes inconsistent target-probability gains than direct token selection optimization.
Limitations
- Information extractors have modest accuracy (around 49-53% Top-1), which the authors attribute to VQGAN generation inaccuracies.
- Codebook-level optimization is computationally intensive and was evaluated only on a 10-category bird subset with 16-token regions.
- Bias analysis is limited to one text-to-image model (DALL-E Mini) and a small set of attributes.
Experimental setup
Datasets: ImageNet VQGAN-generated synthetic ImageNet-class dataset DALL-E Mini generated doctor images
Benchmarks: ImageNet classification via VQGAN synthetic data bird-category codebook optimization test set
Metrics: Top-1/3/5/10 accuracy change in accuracy change in softmax probability mean token frequency Cliff's delta
Baselines: frequency-based token selection random token masking embedding optimization
Builds on: VQGAN Information Bottleneck Gumbel-Softmax VQ-VAE
Closest papers by meaning
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.88)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.88)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations (ICML 2025 · similarity 0.86)
- UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens (NeurIPS 2025 · similarity 0.86)
- Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.