Research map / Label-efficient and robust learning / OOD and anomaly detection
$\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ΔEnergy, an energy-change OOD score obtained by zeroing the top-c image-text cosine similarities, and EBM, a prompt-tuning objective that maximizes a lower bound on ΔEnergy to jointly improve OOD detection and OOD generalization on CLIP.
Problem
Fine-tuning VLMs must both generalize to covariate-shifted closed-set OOD data and detect semantic-shifted open-set OOD classes, but existing methods typically specialize in one of these tasks or are costly.
Approach
Measure the energy change when the maximum image-text cosine similarities are reset to zero (ΔEnergy), which separates closed-set from open-set classes more strongly than MCM or raw energy; during fine-tuning, minimize an energy-change loss on a masked image feature (EBM) that both lower-bounds ΔEnergy and yields domain-consistent Hessians tied to OOD generalization.
Key results
- Zero-shot ΔEnergy reaches 87.10 AUROC vs 85.81 for MCM on ImageNet-1k semantic-shift OOD detection (top-1 AUROC across benchmarks).
- On Setup-II, EBM+ΔEnergy achieves 21.0 average FPR versus 27.2 for ZS CLIP+MCM and 29.0 for GalLoP.
- On Setup-I, EBM reports 63.28% OOD accuracy versus 62.83% for CRoFT and 81.90 AUROC versus 76.30 for CRoFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ΔEnergy instead of MCM for zero-shot OOD detection | CLIP ViT-B/16 · ImageNet-1k ID vs semantic-shifted OOD | AUROC: 85.81 -> 87.10 | Helped |
| Use ΔEnergy instead of MCM for zero-shot OOD detection | CLIP ViT-B/16 · PACS and VLCS vs open-set datasets (average) | FPR95: 27.2 -> 25.9 | Hurt |
| Use ΔEnergy instead of the vanilla Energy score | CLIP ViT-B/16 · ImageNet-1k ID vs semantic-shifted OOD | AUROC: 76.94 -> 87.10 | Helped |
| Add EBM fine-tuning with ΔEnergy scoring | CLIP ViT-B/16 · Setup-I ImageNet-1k | AUROC: 76.30 -> 81.90 (1.9) | Helped |
| Add EBM fine-tuning with ΔEnergy scoring | CLIP ViT-B/16 · Setup-I ImageNet-1k | ID ACC: 84.51 -> 81.52 (0.4) | Hurt |
| Add EBM fine-tuning and use ΔEnergy instead of MCM scoring | CLIP ViT-B/16 · Setup-II PACS and VLCS vs open-set datasets (average) | FPR95: 22.2 -> 21.0 | Helped |
| Add EBM fine-tuning with ΔEnergy scoring | CLIP ViT-B/16 · PACS OOD | OOD ACC: 97.3 (0.1) -> 97.2 (0.1) | No clear effect |
Surprising findings
- Zero-shot ΔEnergy alone does not uniformly beat MCM; on PACS/VLCS it is slightly lower on some datasets, and fine-tuned competitors like CRoFT and DPLCLIP can have higher FPR95 than zero-shot CLIP.
Limitations
- Theoretical guarantees rely on assumptions such as similar sums of non-maximal similarities and small feature-perturbation distances.
- Hyperparameters (tau, c, masking proportion p, lambda0) are selected by search, and the reported ID accuracy is lower than several baselines.
Experimental setup
Datasets: ImageNet-1k ImageNet-A ImageNet-R ImageNet-Sketch ImageNet-V2 PACS VLCS DTD Food101 Caltech101 Texture iNaturalist
Benchmarks: ImageNet-1k open-set split PACS vs open-set VLCS vs open-set ImageNet hard OOD Texture/iNaturalist/Places/SUN conventional OOD
Metrics: AUROC FPR95 OOD accuracy ID accuracy
Baselines: MCM Energy ODIN ReAct CLIPN MSP MaxLogit NegLabel CSP CoOp CoCoOp CLIP-Adapter
Builds on: MCM energy-based OOD score CoOp GL-MCM
Closest papers by meaning
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.92)
- Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations (CVPR 2025 · similarity 0.89)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.89)
- Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.89)
- Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.