Research map / Label-efficient and robust learning / OOD and anomaly detection
Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSKD, a self-distillation framework that periodically selects the best-OOD-detecting checkpoint from training history as a teacher (using pseudo-outliers for selection) and distills its logits and penultimate features to improve OOD detection and ID accuracy.
Problem
OOD detection performance peaks at an intermediate training stage and degrades as the model memorizes atypical samples; existing forgetting-based fixes (UM/UMAP) sacrifice ID generalization.
Approach
Instead of forgetting atypical samples, periodically select the training-history checkpoint with the best AUROC on validation data vs. pseudo-outliers as a self-teacher, and distill its softened logits and penultimate features into the current student, giving uncertainty-embedded soft targets; works as a plug-in with other OOD methods.
Key results
- On CIFAR-10, Energy+PSKD reaches 20.48% FPR95 on far-OOD vs 41.69% for Energy with vanilla training, with ID ACC 95.14%.
- On ImageNet-200, Energy+PSKD lowers average FPR95 from 47.55% to 44.38% and raises AUROC from 86.68% to 87.12%.
- OE+PSKD improves CIFAR-100 near-OOD FPR95 to 48.72% vs 54.82% for OE alone (far-OOD column) and raises ID ACC slightly across benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PSKD to Energy baseline | Energy on CIFAR-10 · CIFAR-10 near- and far-OOD | FPR95: 61.34±4.63 -> 31.67±0.78 (near); 41.69±5.32 -> 20.48±1.30 (far) | Helped |
| Add PSKD to Energy baseline | Energy on CIFAR-100 · CIFAR-100 near- and far-OOD | AUROC: 80.91±0.08 -> 81.45±0.44 (near); 79.77±0.61 -> 82.40±0.52 (far) | Helped |
| Add PSKD to Energy baseline | Energy on ImageNet-200 · ImageNet-200 average OOD | AUROC: 86.68±0.12 -> 87.12±0.15 | Helped |
| Use stationary rather than progressive teacher selection (ablation) | PSKD · CIFAR-10, CIFAR-100, and ImageNet-200 | AUROC: 93.14±0.21 -> 92.82±0.68 (CIFAR-10); 81.93±0.13 -> 81.78±0.06 (CIFAR-100); 87.12±0.15 -> 86.90±0.16 (ImageNet-200) | Hurt |
| Increase teacher selection frequency to 5 selections per epoch (ablation) | PSKD · CIFAR-10, near- and far-OOD average | FPR95: 26.08±1.02 -> 25.79±2.17 | Helped |
| Reduce teacher selection frequency to 1 selection per 5 epochs (ablation) | PSKD · CIFAR-10, near- and far-OOD average | FPR95: 26.08±1.02 -> 26.86±0.91 | Hurt |
| Set feature-distillation weight α to 0 (ablation) | PSKD · CIFAR-10 | AUROC | Hurt |
| Use realistic outliers rather than pseudo-outliers for teacher selection (ablation) | PSKD on ImageNet-200 · ImageNet-200 average OOD | AUROC: 87.12±0.15 -> 87.64±0.18 | Helped |
| Apply PSKD with ViT-B/16 instead of ResNet-18 (ablation) | ImageNet-200 classifier · ImageNet-200 average OOD | AUROC: 93.61±0.19 -> 93.87±0.24 | Helped |
Surprising findings
- Constant or exponential schedules for the distillation weight over-rely on the teacher early and hurt OOD learning, while cosine annealing works best.
- Using realistic outliers for teacher selection only modestly beats pseudo-outliers (ImageNet-200 FPR95 43.93% vs 44.38%).
Limitations
- Gains are smaller on large-scale ImageNet, where the model's intrinsic OOD capability is weaker.
- Teacher update frequency trades OOD performance against extra compute overhead.
- Pseudo-outliers are generated from perturbations of validation images, so their match to realistic OOD distributions is limited; realistic outliers give somewhat better selection.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-200 TinyImageNet-597 MNIST SVHN Textures Places365 SSB-hard NINCO iNaturalist OpenImage-O
Benchmarks: OpenOOD v1.5 CIFAR-10 OpenOOD v1.5 CIFAR-100 OpenOOD v1.5 ImageNet-200
Metrics: FPR95 AUROC ID ACC
Baselines: MSP ODIN Energy LogitNorm SNN T2FNorm MixOE DOE OE UM UMAP
Builds on: Unleashing Mask (UM/UMAP) Knowledge distillation (Hinton et al.) OpenOOD Outlier Exposure (OE) LogitNorm
Closest papers by meaning
- Fast Decision Boundary based Out-of-Distribution Detector (ICML 2024 · similarity 0.88)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.88)
- Can We Ignore Labels in Out of Distribution Detection? (ICLR 2025 · similarity 0.88)
- Beyond Clean Training Data: A Versatile and Model-Agnostic Framework for Out-of-Distribution Detection with Contaminated Training Data (CVPR 2025 · similarity 0.87)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.87)
- A Variational Information Theoretic Approach to Out-of-Distribution Detection (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.