Research map / Label-efficient and robust learning / Recognition and long-tail
Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces unsupervised universal fine-tuning (U²-FT) of CLIP with imprecise class names and OOD samples in unlabeled data, and proposes Universal Entropy Optimization (UEO), which weights entropy minimization/maximization by MSP confidence while tuning the text prompt and visual normalization-layer affine parameters.
Problem
Unsupervised fine-tuning of CLIP when the predefined class-name list only partially matches the label space of unlabeled data, which may contain OOD samples from unknown classes; the goal is to improve both known-class accuracy and OOD detection.
Approach
Use CLIP's maximum softmax probability as sample-level weights: minimize confidence-weighted entropy of confident samples and maximize the entropy of the weighted average prediction (marginal), optimizing only the text prompt and visual normalization affine parameters.
Key results
- With prompt plus normalization-layer tuning on ResNet-50, UEO reaches 71.1% average HAA under closed-set shift, versus 70.2% for the best prompt-only result.
- On DomainNet with ViT-B/16 (+NormLayers), UEO achieves the best HAA under all four category shifts, with 65.8% average HAA.
- UEO reaches 61.1% average accuracy on DomainNet under conventional closed-set fine-tuning, versus 57.6% for CLIP ViT-B/16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual normalization-layer affine optimization to the text prompt (ablation) | UEO on CLIP ResNet-50 · Office, OfficeHome, VisDA-C, and DomainNet average, closed-set | HAA: 70.2 -> 71.1 | Helped |
| Add visual normalization-layer affine optimization (ablation) | UEO on CLIP ResNet-50 · Office, OfficeHome, VisDA-C, and DomainNet average, partial-set | HAA: 69.8 -> 70.6 | Helped |
| Add visual normalization-layer affine optimization (ablation) | UEO on CLIP ResNet-50 · Office, OfficeHome, VisDA-C, and DomainNet average, open-set | HAA: 69.8 -> 70.7 | Helped |
| Add visual normalization-layer affine optimization (ablation) | UEO on CLIP ResNet-50 · Office, OfficeHome, VisDA-C, and DomainNet average, open-partial-set | HAA: 69.1 -> 70.9 | Helped |
| Vary the monotonically decreasing weight function Φ(w) (ablation) | UEO on DomainNet (Sk), open-partial-set · DomainNet (Sk), open-partial-set | ACC: 53.7 to 54.2 | No clear effect |
| Vary the monotonically decreasing weight function Φ(w) (ablation) | UEO on DomainNet (Sk), open-partial-set · DomainNet (Sk), open-partial-set | AUC: 65.9 to 66.4 | No clear effect |
| Use different text prompt initializations instead of “a photo of a” (ablation) | UEO on DomainNet (Sk), open-partial-set · DomainNet (Sk), open-partial-set | ACC: +2.3 to +5.1 | No clear effect |
| Use different text prompt initializations instead of “a photo of a” (ablation) | UEO on DomainNet (Sk), open-partial-set · DomainNet (Sk), open-partial-set | AUC: -1.5 to +1.3 | Mixed |
| Use oracle ID/OOD sample weights in UEO(O) (ablation) | UEO · Unspecified category shift | classification accuracy | Hurt |
| Vary batch size (ablation) | UEO · Unspecified category shift | performance | No clear effect |
Surprising findings
- Under closed-set shift, DANCE with normalization-layer tuning drops sharply on VisDA-C open-set (ACC 61.4%), showing instability of universal domain adaptation baselines.
- Entropy minimization alone (EntMin) does not improve OOD detection and often lowers AUC relative to CLIP.
Limitations
- Evaluated only on image classification; dense prediction tasks such as segmentation and detection are not studied.
- Some metrics gains are small (e.g., ACC/AUC trade-offs), and UEO's AUC is not always best (e.g., second-best on closed-set ACC/AUC).
- Results averaged over only two seeds.
Experimental setup
Datasets: Office OfficeHome VisDA-C DomainNet ImageNet Food101
Benchmarks: Office OfficeHome VisDA-C DomainNet ImageNet Food101
Metrics: ACC AUC HAA OS HOS
Baselines: CLIP EntMin InfoMax DANCE UPL POUF
Builds on: CoOp TENT DANCE POUF UPL
Closest papers by meaning
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.87)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.87)
- What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights (NeurIPS 2024 · similarity 0.86)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.86)
- Targeted Forgetting of Image Subgroups in CLIP Models (CVPR 2025 · similarity 0.86)
- Anchor-based Robust Finetuning of Vision-Language Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.