AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Tune-Up Open-Weight CLIP: Optimization Framework for Self-Supervised Fine-tuning of CLIP

Anant Mehta, Xiyuan Wei, Xingyu Chen, Tianbao Yang

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an optimization framework that self-supervised fine-tunes open-weight CLIP models on existing self-supervised datasets to improve downstream performance without costly retraining from scratch.

Problem

Improving open-weight CLIP models across downstream tasks normally requires training from scratch on billions of samples, which is prohibitively costly.

Approach

Fine-tune pretrained open-weight CLIP models using only self-supervised datasets (no labels), rather than supervised fine-tuning, to improve generalization across downstream tasks.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune with standard OpenCLIP/FastCLIP protocol without OSROpenAI ViT-B/16 · ImageNet & variantsaccuracy: 57.67 -> 54.57 (FastCLIP)Hurt
naive fine-tuning on SigLIP ViT-B/16 degrades stronglySigLIP ViT-B/16 · ImageNet & variantsaccuracy: 63.12 -> 39.22 (FastCLIP)Hurt
add OSR warm-up recovering m and v statistics (ablation)TuneCLIP (HGCL) · ImageNet & variantsaccuracy: 54.91 -> 59.48Improved
additionally recover u statistics in OSR (ablation)TuneCLIP (HGCL) · Mean of IN, Retrieval, DataCompmean score: 60.58 -> 60.70Improved
use hinged global contrastive loss instead of GCL for self-supervised fine-tuning (ablation)OSR + GCL · Flickr retrievalretrievalImproved
use hinged loss in supervised fine-tuning on Flickr30k (ablation)OSR + HGCL · Flickr1kretrievalHurt
add HGCL versus DCL and TNR false negative methods (ablation)TuneCLIP · OpenAI ViT-B/16retrievalImproved
scale fine-tuning data from DFN-12M to DFN-60M (ablation)TuneCLIP · DFN scalemean scoreNo clear effect
fine-tune on noisier CC12M instead of DFN-12M (ablation)TuneCLIP · ImageNet & variantsaccuracy: +0.01 (CC12M) vs +1.69 (DFN12M)No clear effect
CLIP-Refine for SSFT on web dataOpenAI ViT-B/16 · ImageNet & variantsaccuracy: 57.67 -> 55.96Hurt

Experimental setup

Datasets: existing self-supervised datasets

Builds on: CLIP

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy