Research map / Label-efficient and robust learning / Recognition and long-tail
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use independently trained networks instead of other ensemble methods at a fixed test-time budget | Image-classification ensembles · CIFAR-10, CIFAR-100, and ImageNet | Calibrated log-likelihood | Helped |
| Add test-time data augmentation to ensembles | Image-classification ensembles · ImageNet | Calibrated log-likelihood | Helped |
| Add test-time data augmentation to powerful ensembles | Deep Ensembles, Snapshot Ensembles, and Cyclic SGLD · CIFAR-10 and CIFAR-100 | Calibrated log-likelihood | Mixed |
| Add test-time data augmentation | Single-snapshot methods, including VI, K-FAC Laplace, and dropout · CIFAR-10, CIFAR-100, and ImageNet | Calibrated log-likelihood | Helped |
| Use extreme random crops at test time without temperature scaling | Deep Ensembles · ImageNet | Uncalibrated log-likelihood | Hurt |
| Apply temperature scaling before comparing log-likelihood | Ensembling methods · ImageNet | Log-likelihood | Helped |
| Use less extreme random crops for test-time augmentation | Test-time data augmentation · ImageNet | Calibration | Helped |
| Compare models using TACE across different thresholds or bin counts | Uncertainty-estimation evaluation · VGG16BN on CIFAR-100 | TACE ranking consistency | Hurt |
| Use accuracy-confidence curves to compare different models | Misclassification detection · In-domain image classification | Comparability | Hurt |
| Use AUC-ROC or AUC-PR to compare misclassification detection across different models | Misclassification detection · In-domain image classification | Comparability | Hurt |
| Use snapshot-based methods that explore multiple loss-landscape modes rather than single-snapshot methods | Ensembling methods · CIFAR-10, CIFAR-100, and ImageNet | Deep ensemble equivalent | Helped |
| Use Snapshot Ensembles or Cyclic SGLD rather than other methods, except deep ensembles | Ensembling methods · CIFAR-10 and CIFAR-100 | Deep ensemble equivalent | Helped |
Closest papers by meaning
- Uncertainty Quantification and Deep Ensembles (NeurIPS 2021 · similarity 0.89)
- PEP: Parameter Ensembling by Perturbation (NeurIPS 2020 · similarity 0.88)
- SURE: SUrvey REcipes for building reliable and robust deep networks (CVPR 2024 · similarity 0.87)
- FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation (NeurIPS 2022 · similarity 0.86)
- Neural Ensemble Search for Uncertainty Estimation and Dataset Shift (NeurIPS 2021 · similarity 0.86)
- Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.