Research map / Vision-language and multimodal / CLIP and zero-shot

Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Recaptioning and text-supervised pretraining · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Aligns frozen unimodal vision (DINOv2) and text (All-Roberta-Large-v1) encoders by training only lightweight MLP/token projectors with a contrastive loss on a curated 20M image-caption set, matching CLIP zero-shot performance at a fraction of the data and compute.

Problem

Training CLIP-style vision-language models from scratch is computationally expensive and inflexible, and it is unclear whether well-trained unimodal encoders can be connected through simple projections.

Approach

Select semantically similar unimodal encoder pairs using linear CKA on COCO, curate a concept-balanced image-caption dataset via CLIP-embedding prototype retrieval plus high-quality retrieval data, and train only token/MLP projectors (about 11.5M parameters) with a contrastive loss between frozen encoders.

Key results

• DINOv2-ARL projector model reaches 76.3% ImageNet zero-shot accuracy, above OpenAI CLIP ViT-L (75.3%) and LAION-400M CLIP (72.7%), trained on 20M pairs.
• Alignment compute drops to about 400 A100 GPU hours versus 21,845-25,400 for CLIP, a roughly 65-fold reduction; trainable parameters are 11.5M of 670M.
• Image retrieval on Flickr30k reaches 74.1% vs 70.2% for LAION-CLIP ViT-L, while text retrieval matches it (87.5% vs 87.6%).

Measured effects reported in the paper

Surprising findings

• CKA between unimodal encoders correlates strongly (negatively) with the minimum CLIP loss, suggesting encoder similarity predicts alignment ease.
• Projector-only alignment of DINOv2 and a text encoder retains long-context ability, improving retrieval up to about 200-300 tokens beyond CLIP's 77-token limit.

Limitations

• Relies on already strong pretrained unimodal encoders; alignment compute excludes their training cost (DINOv2 used 22k A100 hours).
• Curated LAION-based dataset is noisy in alignment; concept-rich and high-quality data had to be combined to get both zero-shot and retrieval performance.
• Localization gains are evaluated only on Pascal VOC and Pascal Context; fine-grained alignment losses were not explored with projectors.

Experimental setup

Datasets: LAION-400M CC3M CC12M SBU COCO ImageNet

Benchmarks: ImageNet ImageNetV2 Caltech101 Oxford Pets Stanford Cars Flowers102 Food101 FGVC Aircraft SUN397 CUB UCF101 Flickr30k

Metrics: zero-shot accuracy recall@1 mIoU

Baselines: OpenAI CLIP ViT-L LAION-CLIP ViT-L LiT L16L SPARC nllb-clip M-CLIP/XLM-Roberta-Large

Builds on: CLIP DINOv2 Token Projector Centered Kernel Alignment LiT

Closest papers by meaning

• Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.91)
• Assessing and Learning Alignment of Unimodal Vision and Language Models (CVPR 2025 · similarity 0.91)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
• Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.89)
• DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.89)
• CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features (ICLR 2025 · similarity 0.89)