Research map / Vision-language and multimodal / Human and embodied modeling
LG-VQ: Language-Guided Codebook Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LG-VQ, a model-agnostic codebook learning method that aligns VQ codes with pre-trained CLIP text semantics via global alignment, masked text prediction, and word-relationship alignment, improving reconstruction and cross-modal tasks.
Problem
Single-modal VQ codebooks learned from images lack high-level semantics and suffer modal gaps, leading to suboptimal performance on multi-modal downstream tasks such as text-to-image, image captioning, and VQA.
Approach
Use a frozen pre-trained CLIP text encoder as supervision to align codebook features with text globally (InfoNCE), via masked word prediction from visual codes, and by transferring word-pair cosine similarities into code-pair similarities (relationship alignment); the module plugs into existing VQ models.
Key results
- On CelebA-HQ text-to-image, CVQ+LG reaches FID 12.33 vs 13.23 for CVQ and 12.54 for LAFITE.
- On CUB-200 image captioning, LG-VQ improves BLEU4 from 1.29 (VQ-GAN) to 1.69 and ROUGE-L from 33.40 to 34.73.
- On MS-COCO reconstruction, VQ-VAE+LG lowers FID from 86.21 to 79.54 while PSNR drops slightly from 23.55 to 23.40.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LG-VQ to VQ-GAN | VQ-GAN · TextCaps, CelebA-HQ, CUB-200, and MS-COCO reconstruction | FID: 24.08 -> 20.35; 5.66 -> 5.34; 3.63 -> 3.08; 14.45 -> 10.72 | Helped |
| Add LG-VQ to VQ-GAN | VQ-GAN · CelebA-HQ text-to-image | FID: 15.29 -> 12.61 | Helped |
| Add LG-VQ to VQ-GAN | VQ-GAN · CUB-200 image captioning | BLEU4: 1.29 -> 1.69 | Helped |
| Add LG-VQ to VQ-GAN | VQ-GAN · COCO-QA | Accuracy: 37.82 -> 40.97 | Helped |
| Add global semantic alignment loss (ablation) | VQ-GAN · TextCaps reconstruction | FID: 24.08 -> 23.01 | Helped |
| Add global semantic alignment loss (ablation) | VQ-GAN · CUB-200 reconstruction | FID: 3.63 -> 3.39 | Helped |
| Add masked text prediction loss (ablation) | VQ-GAN · TextCaps reconstruction | FID: 24.08 -> 21.54 | Helped |
| Add masked text prediction loss (ablation) | VQ-GAN · CUB-200 reconstruction | FID: 3.63 -> 3.49 | Helped |
| Add relationship alignment loss with masked text prediction (ablation) | VQ-GAN · TextCaps reconstruction | FID: 21.54 -> 20.77 | Helped |
| Add relationship alignment loss with masked text prediction (ablation) | VQ-GAN · CUB-200 reconstruction | FID: 3.49 -> 3.32 | Helped |
| Add global semantic alignment and relationship alignment losses to masked text prediction (ablation) | VQ-GAN · TextCaps reconstruction | FID: 21.54 -> 20.35 | Helped |
| Add global semantic alignment and relationship alignment losses to masked text prediction (ablation) | VQ-GAN · CUB-200 reconstruction | FID: 3.49 -> 3.08 | Helped |
Surprising findings
- Adding LG-VQ improves FID while PSNR gains are marginal or negative on several datasets (e.g., VQ-VAE+LG PSNR slightly lower on TextCaps and CelebA-HQ).
- On CUB-200 captioning, V2L Tokenizer achieves higher CIDEr-D (104.14) than LG-VQ (102.77).
Limitations
- Assumes each word aligns with one code; cannot capture many-to-many word-code relationships.
- Performance on visual text reasoning tasks (captioning, VQA) remains far below dedicated captioning or VQA models.
Experimental setup
Datasets: TextCaps CelebA-HQ CUB-200 MS-COCO COCO-QA RefCOCO
Benchmarks: TextCaps CelebA-HQ CUB-200 MS-COCO COCO-QA RefCOCO
Metrics: FID PSNR BLEU4 ROUGE-L METEOR CIDEr-D Accuracy WUPS Recall@1 MSE
Baselines: VQ-VAE VQ-GAN CVQ VQCT Reg-VQ V2L Tokenizer Unite and Conquer Corgi LAFITE Style ALAE DC-VAE
Builds on: VQ-VAE VQ-GAN CLIP InfoNCE VQCT
Closest papers by meaning
- Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling (CVPR 2024 · similarity 0.91)
- Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text (CVPR 2025 · similarity 0.91)
- Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99% (NeurIPS 2024 · similarity 0.89)
- Learning to Quantize for Training Vector-Quantized Networks (ICML 2025 · similarity 0.88)
- Finite Scalar Quantization: VQ-VAE Made Simple (ICLR 2024 · similarity 0.87)
- Restructuring Vector Quantization with the Rotation Trick (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.