Research map / Learning theory and algorithms / Prediction and decision losses
A Closer Look at Generalized BH Algorithm for Out-of-Distribution Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how calibrated set size affects the generalized BH (g-BH) OOD decision rule, showing its TPR follows a beta distribution, and proposes ensemble g-BH (eg-BH) that integrates multiple empirical p-values from L data splits to improve performance with small calibrated sets.
Problem
The g-BH algorithm for OOD detection relies on empirical p-values computed from a calibrated set of ID data, but the effect of calibrated set size on its performance is unexplored, and small calibration sets degrade detection.
Approach
Derives that the conditional expectation of TPR given the calibrated set follows a beta distribution parameterized by the level and calibration size, and proposes eg-BH which trains L score functions on complementary data partitions, computes L empirical p-values, and integrates them via a generalized power-mean style p-value combination (Theorem 5.1) before applying BH.
Key results
- On CIFAR-10 with Energy and ResNet18 (L=5 setting), eg-BH raises average F1 from 69.45% (g-BH) to 83.81% and average TPR from 73.59% to 84.61%.
- With SVHN as OOD and Energy, g-BH F1 and TPR rise from 52.25%/35.41% at calibration ratio 0.2 to 75.78%/61.51% at ratio 0.8, supporting the claim that larger calibrated sets help.
- eg-BH's advantage over g-BH grows as L increases (smaller calibration set), with the largest gap at L=9 and L=10 across datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ensemble multiple empirical p-values with eg-BH instead of using g-BH | Energy, ResNet18 · CIFAR-10 ID; average across six OOD datasets | F1-score: 69.45 -> 83.81 | Helped |
| Ensemble multiple empirical p-values with eg-BH instead of using g-BH | Energy, ResNet18 · CIFAR-10 ID; average across six OOD datasets | TPR: 73.59 -> 84.61 | Helped |
| Ensemble multiple empirical p-values with eg-BH instead of using g-BH | MSP, ResNet18 · CIFAR-10 ID; average across six OOD datasets | F1-score: 80.84 -> 87.67 | Helped |
| Ensemble multiple empirical p-values with eg-BH instead of using g-BH | MSP, ResNet18 · CIFAR-10 ID; average across six OOD datasets | TPR: 83.39 -> 89.64 | Helped |
| Increase calibrated-set sampling ratio from 0.2 to 0.8 (ablation) | g-BH with Energy, ResNet18 · CIFAR-10 ID / SVHN OOD | F1-score: 52.25 -> 75.78 | Helped |
| Increase calibrated-set sampling ratio from 0.2 to 0.8 (ablation) | g-BH with Energy, ResNet18 · CIFAR-10 ID / SVHN OOD | TPR: 35.41 -> 61.51 | Helped |
| Increase calibrated-set sampling ratio from 0.2 to 0.8 (ablation) | g-BH with Energy, ResNet18 · CIFAR-10 ID / SVHN OOD | FPR: 0.05 -> 0.31 | Hurt |
| Increase the number L of partitions, reducing each calibration subset (ablation) | g-BH and eg-BH with Energy, ResNet18 · CIFAR-10 ID; CIFAR-100, TinyImageNet, Place365, and SVHN OOD | F1-score | Mixed |
Limitations
- Experiments limited to CIFAR-10 as ID data with ResNet18 and WideResNet backbones.
- Theoretical FDR control of eg-BH is stated only for p-values without clear structural dependence; the paper does not verify it empirically.
- Splitting training data into L parts reduces the data available to each score network.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet SVHN Places365 F-MNIST MNIST ImageNet
Benchmarks: CIFAR-100 TinyImageNet SVHN Places365 F-MNIST MNIST
Metrics: TPR FPR F1-score
Baselines: g-BH MSP Energy
Builds on: g-BH algorithm (Ma et al., 2024) Benjamini-Hochberg procedure Energy score MSP
Closest papers by meaning
- A Provable Decision Rule for Out-of-Distribution Detection (ICML 2024 · similarity 0.91)
- A Online Statistical Framework for Out-of-Distribution Detection (ICML 2025 · similarity 0.89)
- Test-Time Linear Out-of-Distribution Detection (CVPR 2024 · similarity 0.84)
- ASAT: Adaptive Scoring and Thresholding with Human Feedback for Robust Out-of-Distribution Detection (NeurIPS 2026 · similarity 0.84)
- Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the Role of Model Complexity (NeurIPS 2025 · similarity 0.84)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.