Research map / Vision-language and multimodal / CLIP and zero-shot
Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
Hao Liu, Wilson Yan, Pieter Abbeel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove L2 normalization in nearest-neighbor code search (ablation) | LQAE · ImageNet linear classification | accuracy: 35.60 -> 30.30 | Hurt |
| remove L2 normalization in nearest-neighbor code search (ablation) | LQAE · miniImageNet 5-way avg | accuracy: 29.04 -> 27.42 | Hurt |
| use randomly initialized instead of pretrained RoBERTa (ablation) | LQAE · miniImageNet 5-way avg | accuracy: 29.04 -> 0.51 | Hurt |
| add entropy regularization 0.5 on quantized codes (ablation) | LQAE · ImageNet linear classification | accuracy: 35.60 -> 30.70 | Hurt |
| remove BERT loss (weight 0.00) (ablation) | LQAE · miniImageNet 2-way avg | accuracy: 53.97 -> 52.45 | No clear effect |
| raise BERT loss weight to 1.00 (ablation) | LQAE · linear vs 5-way few-shot | accuracy: 35.60 -> 36.90 (linear); 29.04 -> 20.93 (5-way) | Mixed |
| vector quantization before decoder input (decoder STE) (ablation) | LQAE · miniImageNet 5-way avg | accuracy: 29.04 -> 30.01 | No clear effect |
| keep only 50% of LQAE codes in the GPT-3 prompt (ablation) | LQAE · miniImageNet 5-way avg | accuracy: 29.04 -> 5.56 | Hurt |
| keep 75% of LQAE codes in the prompt (ablation) | LQAE · miniImageNet 2-way avg | accuracy: 53.97 -> 50.56 | Hurt |
| use smaller GPT-3 (Curie 6.7B instead of Davinci 175B) (ablation) | LQAE · miniImageNet 2-way avg | accuracy: 53.97 -> 46.55 | Hurt |
| use smallest GPT-3 (Babbage 1.3B) (ablation) | LQAE · miniImageNet 5-way avg | accuracy: 29.04 -> 14.70 | Hurt |
| assign VQ-VAE tokens to RoBERTa codebook instead of LQAE training (ablation) | LQAE · miniImageNet 2-way avg | accuracy: 53.97 -> 3.24 | Hurt |
Closest papers by meaning
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.89)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.89)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
