Research map / Vision-language and multimodal / Cross-modal learning
Bridging Vision and Language Spaces with Assignment Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLAP, which trains a single linear layer mapping frozen vision encoder features into frozen LLM word-embedding space using an optimal-transport assignment-prediction loss plus captioning loss, evaluated on zero-shot captioning, VQA and retrieval.
Problem
Linear-transformation approaches for connecting frozen unimodal vision encoders and LLMs suffer from the modality gap and rely on contrastive objectives that poorly bridge embedding spaces, limiting cross-modal alignment quality.
Approach
Use the frozen LLM word embeddings as a fixed central space; assign both image and text representations to words via Sinkhorn-based optimal transport with word-frequency marginals, and train the linear layer to predict each modality's assignment from the other, combined with a captioning loss.
Key results
- On zero-shot MSCOCO captioning, VLAP with CLIP ViT-B/32 and OPT-1.3B reaches 69.9 CIDEr-D versus 54.9 for LiMBeR (CLIP RN50x16, GPT-J), with a 1.3B LM instead of 6B.
- VLAP with BEiT and T5-Base improves CIDEr-D by 14.9% on NoCaps and 29.3% on MSCOCO over LiMBeR with BEiT (per paper text).
- On Visual Dialog T2I retrieval, VLAP improves R@1 by 9.2% over CLIP and 6.1% over FROMAGe; IT2T R@1 improves by 3.5% over prior methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use assignment prediction alongside captioning loss (ablation) | VLAP · MSCOCO | CIDEr-D: 51.7 -> 69.9 | Helped |
| use captioning loss alongside assignment prediction (ablation) | VLAP · MSCOCO | CIDEr-D: 49.6 -> 69.9 | Helped |
| add ITM objective to captioning objective (ablation) | VLAP · MSCOCO | CIDEr-D: 51.7 -> 58.4 | Helped |
| add ITC objective to captioning + ITM objectives (ablation) | VLAP · MSCOCO | CIDEr-D: 58.4 -> 61.8 | Helped |
| add ITM and ITC objectives to captioning + assignment objectives (ablation) | VLAP · MSCOCO | CIDEr-D: 69.9 -> 65.7 | Hurt |
| use fixed LLM word embeddings instead of learnable prototypes (ablation) | VLAP · MSCOCO | CIDEr-D: 20.4 -> 61.8; 43.8 -> 68.0; 38.9 -> 69.9 | Helped |
| use the word-frequency distribution instead of equipartition (ablation) | VLAP · MSCOCO | CIDEr-D: 50.7 -> 61.8; 57.9 -> 68.0; 57.1 -> 69.9; 46.2 -> 67.7 | Helped |
| increase the assignment-loss ratio to 0.6:0.4 relative to captioning loss (ablation) | VLAP · MSCOCO | CIDEr-D: 51.7 -> 69.9 | Helped |
| increase training dataset from CC3M to CC12M (ablation) | VLAP · NoCaps | CIDEr-D: 61.6 -> 62.4 | Helped |
| increase training dataset from CC3M to CC12M (ablation) | VLAP · MSCOCO | CIDEr-D: 69.4 -> 70.1 | Helped |
| increase training dataset from CC3M to CC12M (ablation) | VLAP · VQA2 0-shot | accuracy: 41.1 -> 41.3 | Helped |
| increase training dataset from CC3M to CC12M (ablation) | VLAP · Visual Dialog IT2T | R@1: 21.1 -> 23.0 | Helped |
Surprising findings
- Training with only the captioning loss gives 51.7 CIDEr-D on MSCOCO, while adding the assignment prediction loss raises it to 69.9; ITC and ITM objectives reduced performance when combined with the assignment loss.
- Expanding training data from CC3M to CC12M yields only marginal gains, which the authors attribute to the word-distribution-based transport polytope.
Limitations
- Large performance gap remains against modular methods such as Flamingo and BLIP-2.
- Relies on the word distribution of the training data for the transport polytope, so may fail when training words do not cover real-world scenarios.
- Image-only vision encoder (BEiT) yields weaker captions and VQA failures than CLIP.
Experimental setup
Datasets: CC3M CC12M Visual Dialog VQA2 MSCOCO NoCaps
Benchmarks: MSCOCO NoCaps VQA2 Visual Dialog Visual Relation
Metrics: CIDEr-D CLIP-Score RefCLIP-Score accuracy NDCG MRR Recall@k BLEU-1 METEOR ROUGE
Baselines: Frozen MAGMA LiMBeR FROMAGe ViLBERT CLIP ESPER ClipCap ZeroCap
Builds on: LiMBeR FROMAGe SwAV-style optimal transport clustering LLaVA
Closest papers by meaning
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.91)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.90)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.