Research map / Vision-language and multimodal / CLIP and zero-shot
With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that frozen unimodal encoders can be aligned with only tens of thousands of image-text pairs by adding a multiscale neighborhood-preserving STRUCTURE regularizer and aligning the most representationally similar layers instead of last layers.
Problem
Aligning pretrained unimodal foundation models into a shared multimodal embedding space normally requires tens of millions of paired samples; the paper asks whether tens of thousands of pairs suffice.
Approach
A Jensen-Shannon regularizer that matches multi-hop random-walk similarity distributions between each unimodal latent space and its aligned space (preserving neighborhood geometry), combined with selecting the layer pair with highest mutual-kNN similarity rather than last layers; both plug into existing alignment objectives (Linear, MLP, CSA).
Key results
- Linear + Similar + STRUCTURE improves average zero-shot classification by 68.4% relative and retrieval by 122.8% relative over the last-layer baseline on the main 8-dataset table (abstract reports 51.6% and 91.8% across 24 benchmarks).
- On CIFAR10, the aligned model outperforms CLIP by around 2% while using about 0.02% of CLIP's training data.
- With three in-domain samples per class added to the COCO set, Flowers zero-shot accuracy rises from around 24% to over 95%, above CLIP's 93%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add STRUCTURE to MLP alignment with similar layers (ablation) | MLP alignment · 22 zero-shot classification benchmarks | accuracy: 74.0% | Helped |
| Add STRUCTURE to Linear alignment with similar layers (ablation) | Linear alignment · 22 zero-shot classification benchmarks | accuracy: 68.4% | Helped |
| Add STRUCTURE to CSA with similar layers (ablation) | CSA · 22 zero-shot classification benchmarks | accuracy: 26.8% | Helped |
| Add STRUCTURE to MLP alignment with similar layers (ablation) | MLP alignment · Flickr30 and MS COCO retrieval benchmarks | R@1: 137.0% | Helped |
| Add STRUCTURE to Linear alignment with similar layers (ablation) | Linear alignment · Flickr30 and MS COCO retrieval benchmarks | R@1: 122.8% | Helped |
| Add STRUCTURE to CSA with similar layers (ablation) | CSA · Flickr30 and MS COCO retrieval benchmarks | R@1: 15.9% | Helped |
| Select the most similar layers instead of the last layers (ablation) | Linear alignment · 22 zero-shot classification benchmarks | accuracy: 2.5% | Helped |
| Select the most similar layers instead of the last layers (ablation) | MLP alignment · 22 zero-shot classification benchmarks | accuracy: 4.8% | Helped |
| Select the most similar layers instead of the last layers (ablation) | CSA · 22 zero-shot classification benchmarks | accuracy: 2.0% | Helped |
| Select the most similar layers instead of the last layers (ablation) | Linear alignment · Flickr30 and MS COCO retrieval benchmarks | R@1: 8.6% | Helped |
| Select the most similar layers instead of the last layers (ablation) | MLP alignment · Flickr30 and MS COCO retrieval benchmarks | R@1: 18.3% | Helped |
| Select the most similar layers instead of the last layers (ablation) | CSA · Flickr30 and MS COCO retrieval benchmarks | R@1: 2.7% | Helped |
Surprising findings
- Aligning last layers is often suboptimal; intermediate layers with highest mutual-kNN similarity align better.
- Adding only a few in-domain labeled samples per class lets the low-data model match or exceed CLIP trained on hundreds of millions of pairs.
Limitations
- Remaining performance gap versus CLIP on harder tasks such as Pets (~15% accuracy).
- Only two modalities (image and text) were investigated; multi-modality extension is left to future work.
- Distribution shift between COCO training data and target evaluation domains limits zero-shot transfer.
Experimental setup
Datasets: MS COCO train split LAION-15M (CLIP-score filtered) CLIP benchmark datasets
Benchmarks: STL10 CIFAR10 Caltech101 Food101 CIFAR100 ImageNet Pets Flickr30 MS COCO Flowers
Metrics: zero-shot accuracy R@1 trustworthiness continuity label efficiency utility
Baselines: Linear mapping MLP mapping CSA CLIP ASIF FuseMix
Builds on: CLIP Centered Kernel Alignment (CSA) Platonic representation hypothesis
Closest papers by meaning
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.90)
- Data-Efficient Multimodal Fusion on a Single GPU (CVPR 2024 · similarity 0.89)
- Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.89)
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.89)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.