Research map / Vision-language and multimodal / CLIP and zero-shot
Post-pre-training for Modality Alignment in Vision-Language Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-Refine, a one-epoch post-pre-training method on small image-text data that reduces the CLIP modality gap via random feature alignment and hybrid contrastive-distillation, improving zero-shot accuracy without degradation.
Problem
The modality gap between image and text feature clusters in CLIP limits downstream performance, and existing pre-training or fine-tuning fixes are either too costly or degrade zero-shot transfer.
Approach
Align image and text features to a shared random prior by penalizing distance to sampled reference vectors (RaFA), and train with distillation from the frozen pre-trained CLIP using labels alpha-blended between ground-truth pairs and teacher similarities (HyCD), enabling lightweight post-pre-training.
Key results
- CLIP-Refine raises 12-dataset zero-shot classification average for CLIP ViT-B/32 from 52.74 (pre-trained) to 54.69, while contrastive post-pre-training drops it to 45.75.
- On Flickr30K T→I R@5, CLIP-Refine improves ViT-B/32 from 40.96 to 49.70 (the paper reports the average of 12 datasets and retrieval results across models).
- CLIP-Refine improved zero-shot classification across six pre-trained models, e.g., SigLIP ViT-SO400M/14 from 72.57 to 73.61.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| post-pre-train with CLIP-Refine using RaFA and HyCD | CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 52.74 -> 54.69 | Helped |
| post-pre-train with CLIP-Refine using RaFA and HyCD | CLIP ViT-B/32 · Flickr30K zero-shot I→T | R@5: 44.57 -> 51.54 | Helped |
| post-pre-train with CLIP-Refine | pre-trained CLIP models · Zero-shot classification (12 datasets average) | top-1 accuracy: 52.74 -> 54.69; 68.52 -> 69.05; 71.28 -> 72.59; 72.57 -> 73.61; 77.30 -> 78.29 | Helped |
| add RaFA to HyCD (ablation) | CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 53.13 -> 54.69 | Helped |
| replace RaFA with direct image-text feature alignment loss (ablation) | HyCD on CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 53.13 -> 45.61 | Hurt |
| use standard Gaussian N(0,I) rather than no-randomness N(0,0I) prior (ablation) | CLIP-Refine on CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 53.79 -> 54.69 | Helped |
| use standard Gaussian N(0,I) rather than U(0, 1) prior (ablation) | CLIP-Refine on CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 52.91 -> 54.69 | Helped |
| increase Gaussian prior variance from β=0.01 to β=0.1 (ablation) | CLIP-Refine on CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 54.19 -> 54.32 | Helped |
| increase Gaussian prior variance from β=0.1 to β=1 (ablation) | CLIP-Refine on CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 54.32 -> 54.69 | Helped |
| increase Gaussian prior variance from β=1 to β=100 (ablation) | CLIP-Refine on CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 54.69 -> 53.59 | Hurt |
| post-pre-train with contrastive loss rather than CLIP-Refine | CLIP ViT-B/32 · Zero-shot classification (12 datasets average) | top-1 accuracy: 52.74 -> 45.75 | Hurt |
| post-pre-train with CLIP-Refine rather than contrastive loss | CLIP ViT-B/32 · Zero-shot retrieval | R@5: T→I: 47.32 -> 49.70; I→T: 36.29 -> 51.54 | Mixed |
Surprising findings
- Naive alignment loss (HyCD + L_align) achieves the best modality gap and alignment but severely degrades uniformity and zero-shot accuracy (45.61 average).
- Standard contrastive post-pre-training with small batches (512) degrades zero-shot classification substantially, attributed to catastrophic forgetting from small negative sets.
Limitations
- Gains depend on dataset quality; noisy large datasets like CC3M and CC12M underperform COCO Caption.
- Modality gap is not fully eliminated, remaining at a theoretical lower bound.
- Evaluation focuses on CLIP-style models and a limited set of classification and retrieval tasks.
Experimental setup
Datasets: COCO Caption Flickr8K Flickr30K CC3M CC12M ImageNet
Benchmarks: Aircraft Bird Caltech-101 Car DTD EuroSAT Flower Food ImageNet Pet SUN397 UCF-101
Metrics: top-1 accuracy recall@k modality gap alignment uniformity
Baselines: Pre-trained CLIP Contrastive post-pre-training m2-mix Self-KD HyCD + L_align
Builds on: CLIP CLIP-KD m2-mix random feature regularization
Closest papers by meaning
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.92)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.91)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.91)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.91)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.90)
- Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.