Research map / Label-efficient and robust learning / Noisy labels
SURE: SUrvey REcipes for building reliable and robust deep networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines RegMixup, correctness ranking loss, cosine similarity classifier, SAM and SWA into a single training recipe (SURE) that improves failure prediction and uncertainty-based robustness on CIFAR, long-tailed, noisy-label and corrupted-image benchmarks.
Problem
Uncertainty estimates from deep networks are overconfident and unreliable; existing methods excel only on narrow tasks (failure prediction or OOD detection) and do not generalize uniformly to real-world conditions like corruption, label noise, and long-tailed classes.
Approach
Integrate existing techniques across three aspects—regularization (RegMixup + correctness ranking loss), classifier (cosine similarity classifier) and optimization (SAM + SWA)—so that hard samples get higher entropy and training converges to flat minima, yielding better uncertainty estimates without extra data.
Key results
- On CIFAR100 failure prediction with ResNet-18, SURE reaches 80.49% accuracy and 45.81 AURC versus 75.87% and 69.44 for MSP.
- On Food-101N, SURE attains 88.0% top-1 accuracy, above the prior SOTA Jigsaw-ViT at 86.7%, without task-specific adjustments.
- On CIFAR10-C with DenseNet, average AURC drops from 309 (baseline) to 229 with SURE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine SURE's regularization, classifier, and optimization recipes instead of using FMFP alone | ResNet-18 · CIFAR-100 failure prediction | accuracy: 77.82±0.08 -> 80.49±0.18 | Helped |
| Add uncertainty-aware re-weighting | SURE · CIFAR100-LT, IF=10 | top-1 accuracy: 71.13 -> 73.24 | Helped |
| Add uncertainty-aware re-weighting | SURE · CIFAR10-LT, IF=100 | top-1 accuracy: 83.28 -> 86.93 | Helped |
| Add CSC to the SURE variant with CRL, RegMixup, and FMFP (ablation) | SURE · CIFAR-100 failure prediction | accuracy: 79.17±0.30 -> 80.49±0.18 | Helped |
| Add CRL to the SURE variant with RegMixup, FMFP, and CSC (ablation) | SURE · CIFAR-100 failure prediction | accuracy: 79.88±0.07 -> 80.49±0.18 | Helped |
| Add RegMixup to the SURE variant with CRL, FMFP, and CSC (ablation) | SURE · CIFAR-100 failure prediction | accuracy: 78.24±0.18 -> 80.49±0.18 | Helped |
| Add FMFP (SAM and SWA) to the SURE variant with CRL, RegMixup, and CSC (ablation) | SURE · CIFAR-100 failure prediction | accuracy: 79.42±0.11 -> 80.49±0.18 | Helped |
Surprising findings
- Combining individual techniques does not uniformly help; e.g., CSC alone lowers accuracy on CIFAR100 (74.05 vs 75.87 for MSP).
Limitations
- Gains are largely from combining existing components; the individual contributions are not novel.
- Long-tailed results are not on par with the best specialized methods in all settings (e.g., GLMC+MaxNorm outperforms SURE at some imbalance factors without re-weighting).
Experimental setup
Datasets: CIFAR10 CIFAR100 Tiny-ImageNet CIFAR10-LT CIFAR100-LT Animal-10N Food-101N CIFAR10-C
Benchmarks: CIFAR10 CIFAR100 Tiny-ImageNet CIFAR10-LT CIFAR100-LT Animal-10N Food-101N CIFAR10-C
Metrics: accuracy AURC AUROC FPR95
Baselines: MSP RegMixup CRL SAM SWA FMFP OpenMix GLMC Jigsaw-ViT SSR+
Builds on: RegMixup correctness ranking loss FMFP Sharpness-Aware Minimization Stochastic Weight Averaging Mixup
Closest papers by meaning
- Confidence-based Reliable Learning under Dual Noises (NeurIPS 2022 · similarity 0.90)
- Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (NeurIPS 2017 · similarity 0.90)
- Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers (ICLR 2019 · similarity 0.90)
- Depth Uncertainty in Neural Networks (NeurIPS 2020 · similarity 0.89)
- Improving model calibration with accuracy versus uncertainty optimization (NeurIPS 2020 · similarity 0.89)
- Conservative Uncertainty Estimation By Fitting Prior Networks (ICLR 2020 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.