Research map / Label-efficient and robust learning / OOD and anomaly detection

A Variational Information Theoretic Approach to Out-of-Distribution Detection

ICML 2025 · Label-efficient and robust learning › OOD and anomaly detection · approach: Spurious-aware OOD detection

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a variational information-theoretic loss (KL separation of ID/OOD feature distributions minus an Information Bottleneck term) to derive random OOD feature shaping functions, yielding a piecewise-linear shaping function that beats prior element-wise shaping methods on ImageNet and CIFAR OOD benchmarks.

Problem

Out-of-distribution detection for neural networks currently relies on empirically designed, rule-based feature shaping functions that may not generalize to unseen OOD datasets; there is no principled theory explaining when they work.

Approach

Model the OOD feature as a random variable p(z~|z) and optimize a loss combining a symmetrized KL divergence between ID and OOD feature distributions with an Information Bottleneck regularizer, solved via calculus of variations and gradient descent; the mean of the optimal feature recovers existing shaping functions under different assumed OOD distributions (Gaussian, Laplacian, inverse Gaussian)

Key results

• On ImageNet-1k with ResNet-50, the proposed shaping function achieves FPR95 35.82 and AUROC 89.36, versus 37.97/88.58 for VRA-P and 39.75/88.56 for FS-OPT.
• On CIFAR-10 with DenseNet101, FPR95 drops to 24.08 (AUROC 95.26), better than FS-OPT's 28.90 among element-wise shaping methods.
• On ViT-B-16 and ViT-L-16 (ImageNet), the method reports the lowest FPR95 among compared methods (67.73 and 66.67).

Measured effects reported in the paper

Surprising findings

• Optimal learned OOD features have non-zero conditional standard deviation in all simulations, suggesting randomness helps over deterministic shaping.
• ASH variants sometimes outperform the proposed method, though ASH is not element-wise and is outside the theory's scope.

Limitations

• Theory explains only the element-wise OOD feature, not the scoring function (energy score is used).
• Assumes Gaussian ID features and element-wise independence, which only approximately match empirical distributions.
• Hyperparameters tuned on Gaussian noise (ImageNet) or iSUN validation OOD data, which may favor certain OOD types.

Experimental setup

Datasets: ImageNet-1k CIFAR-10 CIFAR-100 iSUN

Benchmarks: ImageNet-1k OOD benchmark (Species, iNaturalist, SUN, Places, OpenImage-O, ImageNet-O, Texture, MNIST) CIFAR-10/100 OOD benchmark

Metrics: FPR95 AUROC

Baselines: Energy ReAct BFAct VRA-P FS-OPT MSP ODIN ASH DICE

Builds on: Information Bottleneck ReAct Energy-based OOD detection FS-OPT

Closest papers by meaning

• Towards Optimal Feature-Shaping Methods for Out-of-Distribution Detection (ICLR 2024 · similarity 0.90)
• Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection (ICLR 2025 · similarity 0.89)
• Learning to Shape In-distribution Feature Space for Out-of-distribution Detection (NeurIPS 2024 · similarity 0.88)
• Detecting Out-of-Distribution Through the Lens of Neural Collapse (CVPR 2025 · similarity 0.88)
• Fast Decision Boundary based Out-of-Distribution Detector (ICML 2024 · similarity 0.88)
• CASE: Exploiting Intra-class Compactness and Inter-class Separability of Feature Embeddings for Out-of-Distribution Detection (AAAI 2024 · similarity 0.88)