Research map / Label-efficient and robust learning / OOD and anomaly detection

CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection

ICML 2024 · Label-efficient and robust learning › OOD and anomaly detection · approach: Robust fine-tuning for OOD generalization · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CRoFT, a CLIP fine-tuning framework that uses an energy distribution reshaping loss (gradient magnitude of energy scores) plus adversarial covariate-shift feature generation to jointly improve OOD generalization and open-set OOD detection.

Problem

Fine-tuning vision-language pre-trained models can distort general knowledge and hurt robustness to covariate shift, while also failing to detect semantic-shift unseen classes; existing methods address only one of these.

Approach

An energy distribution reshaping (EDR) loss minimizes the squared gradient norm of energy scores on training data; the authors argue this yields domain-consistent Hessians and thus a Hessian-based OOD generalization bound, so open-set detection and closed-set OOD generalization are optimized concurrently, together with an adversarially generated worst-case covariate-shifted feature objective.

Key results

• On Setup-I (ViT-B/16, 16-shot), CRoFT reaches FPR95 52.0 vs 77.3 for zero-shot CLIP, a reduction of up to 25.3 points, with OOD ACC 62.9.
• On Setup-II (16-shot), CRoFT achieves average FPR 38.6 (RN50) and 40.0 (ViT-B/16), about 11% better than the second-best method per the paper.
• CRoFT improves closed-set OOD accuracy over CLIP-Adapter by 4.3% on Setup-II per the paper.

Measured effects reported in the paper

Surprising findings

• Vanilla energy minimization (EnergyMin) reduces FPR95 relative to CLIP but still trails CRoFT by about 15 points, and several prompt/adapter baselines have higher FPR95 than zero-shot CLIP.

Limitations

• Theoretical bound relies on strong assumptions (local Hessian Lipschitzness, small curvature term, worst-case covariate-shift assumption) and the EDR-Hessian link is argued more than rigorously shown.
• Evaluated only on CLIP-based models with lightweight adapters; generalization to other VL-PTMs is claimed but not tested.
• Hyperparameters lambda1 and lambda2 are tuned via search over ranges, with different ranges used per setup.

Experimental setup

Datasets: ImageNet-1K ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2 PACS VLCS Caltech101 DTD Food101

Benchmarks: ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2 PACS VLCS Caltech101 DTD Food101

Metrics: ID accuracy OOD accuracy AUROC FPR95

Baselines: CLIP CoOp CoCoOp CLIP-Adapter Tip-Adapter-F DPLCLIP Bayes-CAL EnergyMin

Builds on: CLIP energy-based OOD detection CLIP-Adapter Tip-Adapter CoOp

Closest papers by meaning

• Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.92)
• $\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization (NeurIPS 2025 · similarity 0.92)
• Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models (NeurIPS 2024 · similarity 0.89)
• Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations (CVPR 2025 · similarity 0.89)
• Towards Calibrated Robust Fine-Tuning of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning (AAAI 2024 · similarity 0.88)