Research map / Label-efficient and robust learning / OOD and anomaly detection

Towards Calibrated Robust Fine-Tuning of Vision-Language Models

NeurIPS 2024 · Label-efficient and robust learning › OOD and anomaly detection · approach: Robust fine-tuning for OOD generalization · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CaRot, a robust fine-tuning method for CLIP that jointly improves OOD accuracy and calibration by combining an orthogonality-constrained multimodal contrastive loss (raising the smallest singular value of ID image covariance) with EMA self-distillation for ID calibration.

Problem

Robust fine-tuning of zero-shot vision-language models improves OOD accuracy but tends to harm confidence calibration on OOD data, and existing calibration methods rarely address OOD calibration without OOD samples.

Approach

Derives a shared upper bound on OOD classification and calibration errors in terms of ID calibration error and the reciprocal of the smallest singular value of the ID input covariance; realizes it with an orthogonality-constrained multimodal contrastive loss (to enlarge the smallest singular value) plus EMA self-distillation (for input-dependent ID calibration).

Key results

• On CLIP ViT-B/16, CaRot reaches 62.55% average shift accuracy vs 61.04% for Lipsum-FT, with average shift ECE 0.0791 vs 0.0914 for Lipsum-FT.
• Ablation on ViT-B/16: the EMA self-distillation term alone lowers OOD ECE from 0.2186 to 0.1698; the full CaRot objective reaches 0.0791.
• On CLIP ViT-L/14, CaRot improves OOD accuracy to 74.13% vs 71.44% for FLYP and reduces OOD ECE to 0.0737 vs 0.1470 for FLYP.

Measured effects reported in the paper

Surprising findings

• Label smoothing gives much smaller OOD calibration gains than EMA self-distillation (OOD ECE 0.1262 vs 0.0791).
• Adding the orthogonality constraint on the text projection matrix degraded both accuracy and calibration, so it is applied only to the visual projection.

Limitations

• Experiments only reach ViT-L scale; the theory's assumptions may hold better for larger models such as ViT-G or ViT-H.
• Validation is limited to CLIP-like vision-language models and image classification.
• A trade-off between ID adaptation and OOD generalization appears as self-distillation strength varies.

Experimental setup

Datasets: ImageNet-1K

Benchmarks: ImageNet-1K ImageNet-V2 ImageNet-R ImageNet-A ImageNet-Sketch ObjectNet ImageNet-C

Metrics: top-1 accuracy expected calibration error

Baselines: ZS FT LP-FT FLYP Lipsum-FT CAR-FT Model Stock ARF

Builds on: FLYP Lipsum-FT LP-FT Born-Again Networks WiSE-FT

Closest papers by meaning

• Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.90)
• Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models (ICML 2025 · similarity 0.89)
• CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.88)
• $\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization (NeurIPS 2025 · similarity 0.88)
• Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance (ICLR 2024 · similarity 0.88)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.87)