Research map / Label-efficient and robust learning / OOD and anomaly detection
AHA: Human-Assisted Out-of-Distribution Generalization and Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AHA, a human-assisted labeling method that picks wild examples from a maximum disambiguation region via noisy binary search, improving both OOD generalization and OOD detection on CIFAR-10 with a few hundred to 1000 labels.
Problem
Wild data mixes ID, covariate OOD, and semantic OOD examples; with a limited labeling budget, it is unclear which examples to label to improve both OOD generalization and OOD detection jointly.
Approach
Label wild examples within the maximum ambiguity threshold region, where the weighted density of semantic OOD equals that of ID plus covariate OOD; find this threshold via noisy binary search (adaptive labeling with confidence-interval shrinking), then label equal numbers on both sides, and train a classifier and a sigmoid-loss OOD detector on the labeled data.
Key results
- On CIFAR-10 with Texture semantic OOD and CIFAR-10-C covariate OOD, AHA reaches 90.51% OOD accuracy and 5.63% FPR95 versus SCONE's 85.56% and 37.15%.
- Averaged over five semantic OOD datasets, AHA improves OOD accuracy from 84.95% (SCONE) to 90.00% and reduces FPR95 from 19.33% to 3.54%.
- On PACS, AHA reports 92.7% average accuracy, above the listed domain generalization baselines (best listed HYPO at 89.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AHA human-assisted labeling instead of SCONE | CIFAR-10 OOD learning · LSUN-C + CIFAR-10-C | OOD Acc.: 84.58 -> 90.69 | Helped |
| Use AHA human-assisted labeling instead of SCONE | CIFAR-10 OOD learning · LSUN-C + CIFAR-10-C | FPR: 10.23 -> 0.02 | Helped |
| Use AHA human-assisted labeling instead of SCONE | CIFAR-10 OOD learning · Texture + CIFAR-10-C | OOD Acc.: 85.56 -> 90.51 | Helped |
| Use AHA human-assisted labeling instead of SCONE | CIFAR-10 OOD learning · Texture + CIFAR-10-C | FPR: 37.15 -> 5.63 | Helped |
| Use AHA human-assisted labeling instead of SCONE | CIFAR-10 OOD learning · SVHN + CIFAR-10-C | FPR: 0.08 -> 0.08 | No clear effect |
| Increase the AHA labeling budget from 100 to 2000 (ablation) | AHA · CIFAR-10-C + Texture | OOD Acc.: 79.77 -> 90.46 | Helped |
| Increase the AHA labeling budget from 100 to 2000 (ablation) | AHA · CIFAR-10-C + Texture | FPR: 14.78 -> 2.04 | Helped |
| Use gradient-based rather than energy sampling score (ablation) | AHA · CIFAR-10-C + Texture | OOD Acc.: 89.58 -> 90.51 | Helped |
| Use margin rather than energy sampling score (ablation) | AHA · CIFAR-10-C + Texture | FPR: 6.37 -> 4.15 | Helped |
| Use AHA instead of top-k sampling (ablation) | human-assisted OOD learning · CIFAR-10-C + Texture, budgets 100–2000 | OOD Acc.: 79.77 vs 85.07; 85.44 vs 88.80; 88.32 vs 89.46; 89.87 vs 90.46 | Helped |
| Vary the loss weight α across 1.0–9.0 (ablation) | AHA training objective · CIFAR-10-C + Texture | OOD Acc.: 90.01; 89.80; 89.73; 89.66; 89.59 | No clear effect |
| Use AHA rather than SCONE | CIFAR-10 OOD learning · CIFAR-10-C, average across 19 covariate shifts | OOD Acc.: 90.29 -> 92.06 | Helped |
Surprising findings
- Labeling the top-k most OOD examples or the near-boundary region performs worse than random-like baselines in some settings, since those regions are dominated by uninformative examples of the other types.
- Labeling the region around the most covariate OOD scores mostly yields semantic OOD examples, so it does not improve OOD generalization as expected.
Limitations
- Relies on the assumption that the human annotator can provide ground-truth labels, including distinguishing covariate from semantic OOD, which is not validated with real annotators.
- Experiments are primarily on CIFAR-10 with synthetic covariate shifts (CIFAR-10-C) and a limited set of semantic OOD datasets; ImageNet-100 results are limited to one semantic OOD dataset.
- Comparison to baselines is not always on equal labeling budgets, since AHA uses human labels while many baselines do not.
Experimental setup
Datasets: CIFAR-10 CIFAR-10-C SVHN Textures LSUN-C LSUN-R Places365 PACS ImageNet-100 iNaturalist
Benchmarks: CIFAR-10 with SVHN/LSUN-C/Texture/Places/LSUN-R semantic OOD and CIFAR-10-C covariate OOD PACS ImageNet-100 with iNaturalist
Metrics: OOD Acc. ID Acc. FPR95 AUROC
Baselines: ERM IRM Mixup VREx EQRM SharpDRO MSP ODIN Energy Mahalanobis ViM KNN
Builds on: SCONE WOODS Outlier Exposure Energy-based OOD detection
Closest papers by meaning
- Bridging OOD Detection and Generalization: A Graph-Theoretic View (NeurIPS 2024 · similarity 0.88)
- OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable? (ICML 2025 · similarity 0.88)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.87)
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.87)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.86)
- Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.