Research map / Label-efficient and robust learning / OOD and anomaly detection
Simplification Is All You Need against Out-of-Distribution Overconfidence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that reducing DNN capacity via knowledge distillation and selectively removing ReLU activations (via gradual Leaky-ReLU annealing) mitigates OOD overconfidence and improves OOD detection, without auxiliary OOD data.
Problem
Deep neural networks often produce overly confident predictions on out-of-distribution inputs (OOD overconfidence), which harms reliability in open-world deployment.
Approach
Attributes OOD overconfidence to model over-complexity and mitigates it by jointly reducing capacity (knowledge distillation into a smaller student) and nonlinearity (progressively replacing low-NRO ReLUs with identity via Leaky ReLU slope annealing during distillation), requiring no auxiliary OOD data.
Key results
- On CIFAR-100 with ResNet-101, Ours (WRN-28) reduces average MMC from 73.84 to 54.50 (ID CIFAR-100 table), outperforming CEDA, ACET, and CCUs.
- Combining Ours (ResNet-18) with FeatureNorm on CIFAR-10 improves average FPR95 from 54.71 to 20.09 and AUROC from 86.53 to 96.13.
- On ImageNet-1k, ReAct + Ours (WRN-50) improves average FPR95 from 54.43 to 47.74 and AUROC from 86.60 to 88.56 (ReAct average FPR95 for WRN-101 goes 52.22 to 37.02).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill ResNet-50 to ResNet-18 and selectively remove ReLUs | ResNet-50 · CIFAR-10 OOD average | MMC: 79.58 -> 76.48 | Helped |
| Distill ResNet-50 to ResNet-18 and selectively remove ReLUs | ResNet-50 · CIFAR-100 OOD average | MMC: 72.12 -> 48.35 | Helped |
| Add selective ReLU removal to knowledge distillation (ablation) | ResNet-50 with KD on CIFAR-10 · CIFAR-10 OOD average | MMC: 76.81 -> 76.48 | Helped |
| Add knowledge distillation to selective ReLU removal (ablation) | ResNet-50 with ReLU removal on CIFAR-10 · CIFAR-10 OOD average | MMC: 78.46 -> 76.48 | Helped |
| Use knowledge distillation instead of pruning for capacity reduction (ablation) | ResNet-50 · CIFAR-10 OOD average | MMC: 77.34 -> 76.81 | Helped |
| Use knowledge distillation instead of pruning for capacity reduction (ablation) | ResNet-101 · CIFAR-10 OOD average | MMC: 78.20 -> 77.01 | Helped |
| Remove ReLUs at 2% (ablation) | ResNet-101 · CIFAR-10 OOD average | MMC: 83.46 -> 77.18 | Helped |
| Remove ReLUs at 5% (ablation) | ResNet-101 · CIFAR-100 OOD average | MMC: 73.84 -> 68.71 | Helped |
| Remove ReLUs at 1% (ablation) | ResNet-101 · CIFAR-10 OOD average | MMC: 83.46 -> 84.46 | Hurt |
| Reduce student capacity to ResNet-10 rather than ResNet-18 (ablation) | CIFAR-100 student model · CIFAR-100 | OOD overconfidence | Hurt |
| Start ReLU removal at different training epochs (ablation) | ResNet-18 · CIFAR-10 and CIFAR-100 | MMC | No clear effect |
| Apply model simplification to MaxLogit | ResNet-50 with MaxLogit · CIFAR-10 OOD average | FPR95: 36.07 -> 26.67 | Helped |
Surprising findings
- Knowledge distillation consistently outperforms network pruning for capacity reduction at comparable parameter counts.
- Reducing capacity too far (e.g., ResNet-10 on CIFAR-100) degrades performance, indicating a moderate-capacity sweet spot.
Limitations
- On CIFAR-10 the base method alone shows no pronounced advantage over confidence-suppression baselines; gains require combination with them.
- Theoretical understanding of why simplification mitigates overconfidence is left for future work.
- Effects of ReLU removal ratios and locations are chosen empirically; some configurations show non-monotonic results.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k ImageNet-100
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k SVHN LSUN-R iSUN Textures TinyImageNet-R SUN Places OpenImage-O
Metrics: mean maximum confidence (MMC) FPR95 AUROC AUPRin
Baselines: ACET CEDA CCUs MSP MaxLogit Energy ReAct FeatureNorm Prune KD FitNet AT
Builds on: knowledge distillation (KD) ReLU removal (Dead-ReLU / Leaky ReLU annealing from [4]) ACET CEDA CCUs
Closest papers by meaning
- Controlling Neural Collapse Enhances Out-of-Distribution Detection and Transfer Learning (ICML 2025 · similarity 0.87)
- Deep Neural Networks Tend To Extrapolate Predictably (ICLR 2024 · similarity 0.87)
- NECO: NEural Collapse Based Out-of-distribution detection (ICLR 2024 · similarity 0.86)
- Detecting Out-of-Distribution Through the Lens of Neural Collapse (CVPR 2025 · similarity 0.86)
- Leveraging Perturbation Robustness to Enhance Out-of-Distribution Detection (CVPR 2025 · similarity 0.86)
- Splitting & Integrating: Out-of-Distribution Detection via Adversarial Gradient Attribution (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.