Research map / Vision-language and multimodal / Cross-modal learning
Unified Lexical Representation for Interpretable Visual-Language Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LexVLA, which aligns a frozen DINOv2 visual encoder and a LoRA-tuned Llama 2 text encoder in a unified lexical vocabulary space via contrastive learning with an overuse penalty, trained on CC-12M.
Problem
Vision-language alignment with CLIP-style latent features is hard to interpret, and learning sparse, interpretable lexical representations for image-text alignment is difficult without ground-truth word supervision and suffers from false activations of meaningless tokens.
Approach
Use single-modal pretrained models (frozen DINOv2 for vision, LoRA-tuned Llama 2 as an in-context lexical predictor for text) with a shared vocabulary but separate codebooks (text codebook frozen, image codebook initialized from it), and an overuse penalty that penalizes frequently activated tokens more strongly than FLOPs loss.
Key results
- On MSCOCO zero-shot image-to-text retrieval, LexVLA trained on 12M data reaches R@1 55.4 and R@10 88.3, outperforming CLIP trained on 0.4B and 1.1B data at R@10 of 84.6 and 85.6 respectively (CLIP 1.1B R@1 53.4)
- On PatchDis zero-shot patch-level mIoU on MSCOCO 2017, LexVLA scores 36.3 versus 5.3 for CLIP and 12.6 for VDR
- Trainable parameters total 109M, with the final LexVLA activating on average 1081 dimensions
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LexVLA instead of STAIR | Zero-shot cross-modal retrieval · Flickr30k image-to-text, R@1 | R@1: 81.2 -> 83.9 | Helped |
| Use LexVLA instead of STAIR | Zero-shot cross-modal retrieval · MSCOCO image-to-text, R@1 | R@1: 57.7 -> 55.4 | Hurt |
| Use the LLM lexical predictor instead of caption bag-of-words (ablation) | LexVLA · MSCOCO image-to-text, R@1 | R@1: 17.9 -> 55.4 | Helped |
| Use DINOv2 instead of the original CLIP visual model (ablation) | LexVLA · MSCOCO image-to-text, R@1 | R@1: 51.8 -> 55.4 | Helped |
| Use the overuse penalty instead of FLOPs loss (ablation) | LexVLA · MSCOCO image-to-text, R@1 | R@1: 56.2 -> 55.4 | Hurt |
| Use the overuse penalty instead of FLOPs loss (ablation) | LexVLA · MSCOCO text-to-image, R@1 | R@1: 39.0 -> 39.8 | Helped |
| Use value thresholding instead of top-k thresholding at k=512 (ablation) | LexVLA · MSCOCO text-to-image, R@1 | R@1: 38.1 -> 39.8 | Helped |
| Use value thresholding instead of top-k thresholding at k=512 (ablation) | LexVLA · Flickr30k image-to-text, R@1 | R@1: 84.5 -> 83.9 | Hurt |
| Use LexVLA instead of VDR | Patch-level discrimination · MSCOCO 2017 validation | mIoU: 12.6 -> 36.3 | Helped |
| Use DINOv2 instead of the original CLIP visual model (ablation) | LexVLA patch-level discrimination · MSCOCO 2017 validation | mIoU: 13.9 -> 36.3 | Helped |
Surprising findings
- FLOPs loss, a standard sparsity regularizer, falsely activates semantically unrelated tokens such as '@' and 'contemporary' that act as latent dimensions
- A bag-of-words lexical predictor performs very poorly (MSCOCO image-to-text R@1 17.9), far below the LLM-based lexical predictor
Limitations
- Vocabulary derived from Llama 2 tokenizer splits words into subword tokens, so lexical entries are not complete words
- Word-level vocabulary learned from scratch or via projector performed poorly
- Trained on a 9.2M-pair subset of CC-12M due to download failures
Experimental setup
Datasets: CC-12M Flickr30k MSCOCO 2017
Benchmarks: Flickr30k MSCOCO MSCOCO 2017 (PatchDis)
Metrics: R@1 R@5 R@10 mIoU
Baselines: CLIP FILIP DeCLIP SLIP ProtoCLIP VDR STAIR CLIP-BERT
Builds on: CLIP FLOPs regularization LoRA DINOv2 Llama 2
Closest papers by meaning
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.89)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.88)
- Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models (ICML 2024 · similarity 0.88)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.88)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.88)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.