Research map / Label-efficient and robust learning / OOD and anomaly detection
A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification
Paul F Jaeger, Carsten Tim Lüth, Lukas Klein, Till J. Bungert
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate confidence scoring methods across varied datasets and realistic distribution shifts using AURC | Confidence scoring functions for image classifiers · FD-Shifts benchmark | AURC | Improved |
| Use MCD-MSR rather than ConfidNet | CNN classifier · CIFAR-10 and CIFAR-100 iid | AURC | Improved |
| Use OoD-D methods rather than softmax baselines | Confidence scoring functions · Non-semantic new-class shifts | AURC | Improved |
| Apply OoD-D methods across nuanced covariate shifts instead of their intended new-class scenarios | OoD-D confidence scoring functions · Covariate shifts | AURC | Hurt |
| Use expected entropy rather than mutual information | MCD uncertainty measures · IID test sets | AURC | Improved |
| Expect mutual information to outperform expected entropy on distribution shifts | MCD uncertainty measures · Distribution shifts | AURC | No clear effect |
| Use 32-bit rather than 64-bit precision for MSR logits | ViT classifier · CIFAR-10 | AURC * 10^3: 0.950 -> 3.480 | Hurt |
| Cast MSR logits to 64-bit rather than 32-bit precision | ViT classifier · BREEDS | AURC * 10^3: 4.559 -> 1.893 | Improved |
| Use ViT rather than CNN | Classifier with confidence scoring · Most datasets | AURC | Improved |
| Use ViT rather than CNN | Classifier with confidence scoring · iWildCam | AURC | Hurt |
Closest papers by meaning
- Interpretable Failure Detection with Human-Level Concepts (AAAI 2025 · similarity 0.90)
- Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models (ICLR 2024 · similarity 0.85)
- Typicalness-Aware Learning for Failure Detection (NeurIPS 2024 · similarity 0.85)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.84)
- Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled Yet Hard-to-Learn Samples in Noisy Data (AAAI 2025 · similarity 0.84)
- Long-Tailed Anomaly Detection with Learnable Class Names (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
