Research map / Vision-language and multimodal / Human and embodied modeling
Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VQCT, which transfers a pretrained language-model codebook (CLIP, filtered to adjective and noun tokens via WordNet part-of-speech) through a graph-convolution codebook transfer network to mitigate codebook collapse in vector-quantized image modeling.
Problem
Vector-quantized image modeling suffers from codebook collapse when the codebook is learned from scratch in a code-independent manner, leaving many code vectors never updated.
Approach
Instead of learning codes from scratch, build adjective and noun codebooks from a pretrained language-model embedding space (filtered by part-of-speech) and generate the VQIM codebook with a GCN over an adjective-noun modifying graph, so that codes are updated cooperatively through their semantic relationships.
Key results
- On CUB-200, VQCT reaches FID 2.13 and PSNR 25.35 versus 3.63 and 22.19 for the VQ-GAN baseline.
- On CelebA-HQ semantic image synthesis, VQCT achieves FID 14.47 versus 17.42 for VQ-GAN and 15.34 for Reg-VQ.
- Codebook utilization on CUB-200 rises from about 27.90% (VQ-GAN) to about 42.20% with VQCT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer a pretrained language codebook instead of using random initialization (ablation) | VQCT · CUB-200 | FID: 3.45 -> 2.13 | Helped |
| use a GCN rather than an MLP transfer network with separate adjective and noun codebooks (ablation) | VQCT · CUB-200 | FID: 2.33 -> 2.13 | Helped |
| model adjective and noun codebooks separately rather than concatenate them, using GCN transfer (ablation) | VQCT · CUB-200 | FID: 3.71 -> 2.13 | Helped |
| apply codebook transfer to VQ-VAE (ablation) | VQ-VAE · CUB-200 | FID: 54.92 -> 24.39 | Helped |
| apply codebook transfer to VQ-GAN (ablation) | VQ-GAN · CUB-200 | FID: 3.63 -> 2.13 | Helped |
| use a CLIP rather than GloVe pretrained codebook (ablation) | VQCT · CUB-200 | FID: 3.43 -> 2.13 | Helped |
| transfer a pretrained codebook rather than use VQ-GAN baseline (ablation) | VQCT · CUB-200 | FID: 3.63 -> 2.13 | Helped |
| use codebook transfer for semantic image synthesis rather than Reg-VQ | VQCT · CelebA-HQ | FID: 15.34 -> 14.47 | Helped |
Surprising findings
- CLIP-derived codebooks outperformed GloVe-derived ones, which the authors attribute to multimodal alignment pretraining.
Limitations
- Gains on some metrics are not uniform; VQCT does not beat all baselines on every dataset in Table 1 (e.g., CelebA-HQ l1/l2 and MS-COCO FID vs. CVQ).
- Image synthesis evaluation is a simple single-experiment check using LDM on CelebA-HQ only.
Experimental setup
Datasets: ADE20K CelebA-HQ CUB-200 MS-COCO WordNet
Benchmarks: ADE20K CelebA-HQ CUB-200 MS-COCO
Metrics: FID PSNR l1 l2
Baselines: VQ-VAE VQ-GAN Gumbel-VQ CVQ Reg-VQ SPAE
Builds on: VQ-VAE VQ-GAN CLIP SPAE
Closest papers by meaning
- LG-VQ: Language-Guided Codebook Learning (NeurIPS 2024 · similarity 0.91)
- Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text (CVPR 2025 · similarity 0.89)
- Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99% (NeurIPS 2024 · similarity 0.88)
- Learning to Quantize for Training Vector-Quantized Networks (ICML 2025 · similarity 0.87)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.85)
- Channel-wise Vector Quantization (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.