Research map / Architectures and efficiency / Pruning and efficient tuning
Tune-Up Open-Weight CLIP: Optimization Framework for Self-Supervised Fine-tuning of CLIP
Anant Mehta, Xiyuan Wei, Xingyu Chen, Tianbao Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an optimization framework that self-supervised fine-tunes open-weight CLIP models on existing self-supervised datasets to improve downstream performance without costly retraining from scratch.
Problem
Improving open-weight CLIP models across downstream tasks normally requires training from scratch on billions of samples, which is prohibitively costly.
Approach
Fine-tune pretrained open-weight CLIP models using only self-supervised datasets (no labels), rather than supervised fine-tuning, to improve generalization across downstream tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with standard OpenCLIP/FastCLIP protocol without OSR | OpenAI ViT-B/16 · ImageNet & variants | accuracy: 57.67 -> 54.57 (FastCLIP) | Hurt |
| naive fine-tuning on SigLIP ViT-B/16 degrades strongly | SigLIP ViT-B/16 · ImageNet & variants | accuracy: 63.12 -> 39.22 (FastCLIP) | Hurt |
| add OSR warm-up recovering m and v statistics (ablation) | TuneCLIP (HGCL) · ImageNet & variants | accuracy: 54.91 -> 59.48 | Improved |
| additionally recover u statistics in OSR (ablation) | TuneCLIP (HGCL) · Mean of IN, Retrieval, DataComp | mean score: 60.58 -> 60.70 | Improved |
| use hinged global contrastive loss instead of GCL for self-supervised fine-tuning (ablation) | OSR + GCL · Flickr retrieval | retrieval | Improved |
| use hinged loss in supervised fine-tuning on Flickr30k (ablation) | OSR + HGCL · Flickr1k | retrieval | Hurt |
| add HGCL versus DCL and TNR false negative methods (ablation) | TuneCLIP · OpenAI ViT-B/16 | retrieval | Improved |
| scale fine-tuning data from DFN-12M to DFN-60M (ablation) | TuneCLIP · DFN scale | mean score | No clear effect |
| fine-tune on noisier CC12M instead of DFN-12M (ablation) | TuneCLIP · ImageNet & variants | accuracy: +0.01 (CC12M) vs +1.69 (DFN12M) | No clear effect |
| CLIP-Refine for SSFT on web data | OpenAI ViT-B/16 · ImageNet & variants | accuracy: 57.67 -> 55.96 | Hurt |
Experimental setup
Datasets: existing self-supervised datasets
Builds on: CLIP
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.89)
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.88)
- ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization (AAAI 2024 · similarity 0.88)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.88)
- Captured by Captions: On Memorization and its Mitigation in CLIP Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
