Research map / Vision-language and multimodal / CLIP and zero-shot

Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder

NeurIPS 2026 · Vision-language and multimodal › CLIP and zero-shot · approach: CLIP training recipes and data curation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that contrastive (InfoNCE) post-training of CLIP works when the temperature is small (tau=0.01), and proposes ComCLIP, a single-epoch recipe that freezes the text encoder and combines tempered contrastive loss, MSE anchoring, and DINOv2 relational distillation.

Problem

Recent CLIP post-training work claims the standard contrastive loss causes catastrophic forgetting under small batches, so methods avoid it; the paper asks whether this failure is really due to batch size or to the contrastive temperature, and how to refine CLIP's vision encoder while keeping it dro

Approach

For InfoNCE, forgetting is driven mainly by a large temperature (tau≈0.07), not by few negatives: a small tau (0.01) makes gradients on already-separated pairs vanish, so contrastive post-training refines the model. Combined with a frozen text encoder, MSE anchoring to the original CLIP, and row-wise relational KD from DINOv2, this yields a drop-in vision encoder.

Key results

• On ViT-B/16 (3 seeds), ComCLIP reaches 63.00 zero-shot vs 62.87 for CLIP-Refine (not significant) and linear probe 48.99 vs 42.28.
• On ViT-L/14, ComCLIP improves MMVP over CLIP-Refine (24.20 vs 19.01, p=0.025, 3 seeds, both on CC3M).
• With tau=0.07 contrastive post-training drops 12-dataset zero-shot from 61.82 to 57.90, while tau=0.01 raises it to 62.90.

Surprising findings

• Temperature magnitude, not batch size, explains the reported catastrophic forgetting of InfoNCE post-training; batch size 2048 still does not match small-tau gains.
• CLIP-Refine reduces linear-probe accuracy below the original CLIP on ViT-B/16 (42.28 vs 44.70).

Limitations

• Downstream VLM evaluation uses only LLaVA-1.5-7B with frozen projector; ComCLIP yields no net gain without re-alignment.
• CLIP-Refine remains stronger on image-text retrieval, and MMVP gains rest on only 135 pairs with large seed variance.
• Temperature analysis covers InfoNCE only and does not directly apply to KL-based self-distillation or SigLIP's sigmoid loss.

Experimental setup

Datasets: CC3M CC12M COCO Caption ImageNet-1K

Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 Caltech101 FER2013 OxfordPets DTD RESISC45 EuroSAT PCAM ImageNet-Sketch ImageNet-O

Metrics: top-1 accuracy R@1 linear probe accuracy relative change

Baselines: OpenAI CLIP CLIP-Refine KUEA

Builds on: CLIP CLIP-Refine KUEA relational knowledge distillation

Closest papers by meaning

• Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
• CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.90)
• Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.89)
• A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.88)
• Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)