Research map / Vision-language and multimodal / Human and embodied modeling
Towards Improved Text-Aligned Codebook Learning: Multi-Hierarchical Codebook-Text Alignment with Long Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TA-VQ, which generates long VLM captions (via ShareGPT4V) and aligns multi-granularity (word, phrase, sentence) text to a hierarchical VQ codebook using a sampling-based Wasserstein alignment loss.
Problem
Text-aligned codebook learning for VQ image tokenizers is limited because existing image captions are too brief to provide fine-grained semantic supervision for codebook-text alignment.
Approach
Replace short captions with long VLM-generated descriptions, split them into word/phrase/sentence granularities encoded by BERT, use a hierarchical encoder that yields three code levels matched to these granularities, and align them with a sampling-based Wasserstein (Sinkhorn) loss that reduces computation.
Key results
- TA-VQ reduces CelebA-HQ reconstruction FID to 5.03 with VQ-GAN backbone and TA-CVQ reaches 4.71 vs LG-CVQ 4.90 on CelebA-HQ.
- On CUB-200 image captioning, TA-VQ achieves CIDEr-D 109.42 vs LG-VQ 102.77.
- Sampling-based alignment reduces training time per epoch from 2,065 s to 1,930 s on CUB-200.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sentence-level alignment loss (Ls) to VQ-GAN baseline (ablation) | VQ-GAN · CUB-200 | FID: 5.51 -> 4.89 | Helped |
| add word and phrase alignment losses without sentence loss (Lw+Lp) (ablation) | VQ-GAN · CUB-200 | FID: 5.51 -> 4.75 | Helped |
| add all three alignment losses (Lw, Lp, Ls) (ablation) | VQ-GAN · CUB-200 | FID: 5.51 -> 4.60 | Helped |
| remove multi-granularity text encoding (align full long text directly) (ablation) | TA-VQ · CUB-200 | FID: 4.60 -> 4.91 | Hurt |
| sampling-based alignment instead of full Wasserstein over all codes (ablation) | TA-VQ · CUB-200 training time | sec/epoch: 2,065 -> 1,930 sec/epoch | Helped |
| long VLM-generated text with multi-granularity alignment on VQ-GAN backbone | VQ-GAN · CelebA-HQ, CUB-200, MS-COCO reconstruction | FID: CelebA-HQ 5.66->5.03; CUB-200 5.31->4.60; MS-COCO 14.45->10.32 | Helped |
| long text alignment applied to CVQ backbone | CVQ · CelebA-HQ, CUB-200, MS-COCO reconstruction | FID: CelebA-HQ 5.19->4.71; CUB-200 4.64->4.03; MS-COCO 9.94->9.65 | Helped |
| long text alignment on VQ-GAN backbone vs LG-VQ | VQ-GAN · CUB-200 image captioning | CIDEr-D: 102.77 -> 109.42 | Helped |
| long text alignment on VQ-GAN backbone | VQ-GAN · COCO-QA VQA | Accuracy: 40.97 -> 41.56 | Helped |
| long text alignment on VQ-GAN backbone | VQ-GAN · CelebA-HQ text-to-image (VQ-diffusion) | FID: 12.61 -> 11.97 | Helped |
| long text alignment on VQ-GAN backbone | VQ-GAN · RefCOCO visual grounding | Accuracy(0.5): 9.62 -> 10.17 | Helped |
| long text alignment on VQ-GAN backbone | VQ-GAN · CelebA-HQ semantic synthesis / image completion / unconditional generation | FID: semantic synthesis 11.46->10.74; completion 8.14->8.04; unconditional 9.1->8.8 (vs LG-VQ) | Helped |
Limitations
- Gains over LG-VQ are modest (e.g., 5.03 vs 5.34 FID on CelebA-HQ, 10.32 vs 10.72 on MS-COCO; some tasks close)
- Long-text generation depends on a single VLM (ShareGPT4V) and its quality is not analyzed
- Efficiency evaluated only on a single-batch setting on CUB-200
Experimental setup
Datasets: CelebA-HQ CUB-200 MS-COCO RefCOCO COCO-QA
Benchmarks: CelebA-HQ CUB-200 MS-COCO RefCOCO COCO-QA
Metrics: FID accuracy WUPS BLEU4 ROUGE-L METEOR CIDEr-D cosine similarity
Baselines: VQ-GAN CVQ LG-VQ VQCT LG-CVQ DC-VAE Corgi LAFITE
Builds on: VQ-GAN CVQ LG-VQ VQCT ShareGPT4V
Closest papers by meaning
- LG-VQ: Language-Guided Codebook Learning (NeurIPS 2024 · similarity 0.91)
- Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling (CVPR 2024 · similarity 0.89)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.86)
- From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities (ICLR 2025 · similarity 0.86)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.86)
- Aligning Text to Image in Diffusion Models is Easier Than You Think (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.