Research map / Vision-language and multimodal / CLIP and zero-shot
Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aligns CLIP's visual encoder to DINOv2 via kernel-space (polynomial kernel) matrix matching on ImageNet images, with an L2 regularizer to keep CLIP's text alignment, improving zero-shot recognition, localization, and LLaVA/OpenFlamingo performance.
Problem
CLIP's visual encoder has limited fine-grained perception, which degrades downstream MLLMs; existing fixes either break compatibility with CLIP's frozen text encoder or require costly retraining.
Approach
Align CLIP image-embedding kernel matrices with DINOv2's kernel matrices via a mini-batch-friendly pairwise kernel-matching loss plus an L2 penalty to the original CLIP embeddings, so the visual encoder improves while the frozen text encoder remains compatible.
Key results
- Kernel alignment raises average zero-shot accuracy over 12 datasets for CLIP ViT-L-14 from 65.26 to 66.54 (+1.28%) and ViT-B-16 from 61.22 to 62.04.
- On LLaVA-1.5-7B with ViT-L-14-336, LoRA fine-tuning with the aligned encoder gives 61.70 average vs 60.06 baseline (+1.66%), with localization up ~3.05% on average.
- On OpenFlamingo-3B, replacing the CLIP encoder with the aligned version without LLM tuning improves the average VQA-style score at 0, 4, and 16 shots (e.g., 42.15 to 42.84 at 0-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use kernel-based alignment instead of feature-based alignment | CLIP ViT-L-14 · Average Zero-shot | accuracy: 65.73 -> 66.54 | Helped |
| Align CLIP ViT-L-14 with DINOv2 | CLIP ViT-L-14 · Average Zero-shot | accuracy: 65.26 -> 66.54 | Helped |
| Align CLIP ViT-L-14-336 with DINOv2 | CLIP ViT-L-14-336 · Average Zero-shot | accuracy: 66.10 -> 67.13 | Helped |
| Align CLIP ViT-L-14 with DINOv2 | CLIP ViT-L-14 · CLEVR distance (zero-shot) | accuracy: 20.21 -> 23.39 | Helped |
| Align CLIP ViT-L-14 with DINOv2 | CLIP ViT-L-14 · MSCOCO local probing | macro-averaged recall: 46.40 -> 47.44 | Helped |
| Replace the LLaVA ViT-L vision encoder with its aligned version | LLaVA-1.5-7B · LLaVA vision-language benchmarks | Average: 48.50 -> 48.87 | Helped |
| Replace the OpenFlamingo visual encoder with aligned CLIP | OpenFlamingo-3B (Instruct) · OpenFlamingo (0 shots) | Average: 42.15 -> 42.84 | Helped |
| Align CLIP ViT-L-14 with DINOv2 | CLIP ViT-L-14 · MMVP-VLM | accuracy: 17.78 -> 20.74 | Helped |
| Replace LLaVA's visual encoder with aligned CLIP ViT-L-14-336 | LLaVA-1.5-7B · MMVP | accuracy: 23.33 -> 24.67 | Helped |
| Use a linear projection of DINOv2 features into CLIP space instead of kernel alignment | CLIP ViT-L-14 · Average Zero-shot | accuracy: 66.54 vs 54.07 | Hurt |
Surprising findings
- Linear projection of DINOv2 features into CLIP space, and feature-based alignment, often hurt zero-shot accuracy substantially (e.g., projection drops average to 54.07 for ViT-L-14).
- On the MMVP-VLM benchmark the aligned encoder improves accuracy by only a few points, and the MMVP LLaVA gain without LLM tuning is small (23.33 to 24.67).
Limitations
- Alignment uses a relatively small dataset (ImageNet-1K) and only small MLLMs (LLaVA-1.5-7B, OpenFlamingo-3B) were evaluated; larger scales (e.g., 70B) remain untested.
- Only CLIP-DINOv2 pairing is studied in depth; other MLLM vision encoders and vision-centric models are not explored.
- Gains are modest in several benchmarks (e.g., about 1% average zero-shot improvement and small gains on MMVP-VLM) and attention maps change little.
Experimental setup
Datasets: ImageNet-1K LLaVA-1.5 data mixture CLIP-Benchmark datasets MMVP-VLM MMVP
Benchmarks: ImageNet CIFAR10 CIFAR100 Caltech101 FER2013 OxfordPets DTD RESISC45 EuroSAT PCAM ImageNet-S ImageNet-O
Metrics: zero-shot accuracy recall@K macro-averaged recall CIDEr VQA accuracy ROC AUC
Baselines: CLIP linear projection feature-based alignment DIVA AM-RADIO Additive-MoF
Builds on: CLIP DINOv2 DIVA LLaVA OpenFlamingo
Closest papers by meaning
- Unlocking Fine-Grained Perception in CLIP via Structurally-Aware Latent Masked Modeling (NeurIPS 2026 · similarity 0.91)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.91)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.91)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.