Research map / Label-efficient and robust learning / Recognition and long-tail
Training Deep Neural Networks with Virtual Smoothing Classes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Virtual Smoothing (VS) labels, which move part of the target confidence onto extra virtual classes instead of spreading it over real incorrect classes, improving accuracy, calibration, distillation, adversarial training and OOD detection.
Problem
Training with one-hot labels causes overconfidence, while label smoothing harms logit similarity information and can hurt calibration when high accuracy requires high confidence.
Approach
Add V virtual classes that receive total confidence alpha uniformly, so the real classes keep zero confidence on incorrect classes; the virtual classes act as adversaries that cluster all samples to regularize overconfidence, and softmax is applied only to real-class logits at inference so the correct-class probability can still approach 1.
Key results
- On CIFAR100 with ResNeXt-29, VS labels improve accuracy by 3.3% over one-hot versus 1.64% for label smoothing.
- On CIFAR10 with AT (WRN-34-10), VS labels raise Auto-Attack+ robustness from 51.88% to 53.16%.
- VS-trained ResNet-18 achieves 0.0154 ECE on CIFAR10 at its best alpha, while LS reaches 0.3466 at its best alpha.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply virtual smoothing labels instead of one-hot labels | ResNet-18 and ResNeXt models · SVHN, CIFAR10, CIFAR100, Tiny-200 | accuracy: + 0.67; + 0.74; + 2.62; + 1.76 (ResNet-18); + 0.67; + 1.41; + 3.3; + 1.76 (ResNeXt) | Helped |
| Use virtual smoothing instead of label smoothing | ResNet-18 and ResNeXt models · SVHN, CIFAR10, CIFAR100, Tiny-200 | accuracy: 96.02 vs 95.74; 95.30 vs 94.99; 78.05 vs 77.56; 65.86 vs 65.00; 96.79 vs 96.51; 95.27 vs 94.88; 79.84 vs 78.18; 66.19 vs | Helped |
| Apply virtual smoothing labels | ImageNet classifiers · ImageNet | accuracy: 70.74 vs 70.69; 78.39 vs 78.31; 79.08 vs 79.01; 79.70 vs 79.45 | Helped |
| Apply virtual smoothing rather than label smoothing | ResNeXt models · CIFAR10, CIFAR100, ImageNet | ECE: 0.0154 vs 0.3466; 0.0302 vs 0.6947; 0.0374 vs 0.1056 | Helped |
| Train the teacher with virtual smoothing labels | knowledge distillation students · SVHN, CIFAR10, CIFAR100, Tiny-200 | accuracy: 96.62; 95.21; 78.61; 66.37 | Helped |
| Use virtual smoothing labels in adversarial training | AT, TRADES, AWP, and RST on WRN-34-10 · CIFAR10 | AA+ accuracy: 51.88 -> 53.16; 54.01 -> 54.80; 56.20 -> 57.34; 59.57 -> 59.94 | Helped |
| Use virtual smoothing labels in adversarial training | AT, TRADES, and AWP on WRN-34-10 · CIFAR100 | AA+ accuracy: 28.13 -> 29.00; 28.64 -> 29.19; 29.63 -> 30.71 | Helped |
| Use virtual smoothing instead of label smoothing in AT | WRN-34-10 · CIFAR10 | AA+ accuracy: 52.79 -> 53.16 | Helped |
| Use virtual smoothing instead of label smoothing in AT | WRN-34-10 · CIFAR100 | AA+ accuracy: 28.68 -> 29.00 | Helped |
| Train the robust teacher with virtual smoothing | adversarial robust distillation students · CIFAR10 and CIFAR100 | Auto-Attack+ accuracy: 49.79; 26.68; 49.90; 27.52 | Helped |
| Add virtual smoothing to OE or SSL | OOD detection models · CIFAR10 and CIFAR100 | AUC: OE: 98.93 -> 99.27; SSL: 99.03 -> 99.39; OE: 91.42 -> 92.84; SSL: 91.14 -> 93.00 | Helped |
| Vary the number of VS classes to V=10 or V=20 (ablation) | ResNet-18 or ResNeXt-29 · CIFAR10 | accuracy: 94.56 -> 95.3; 93.86 -> 95.49 | Helped |
Surprising findings
- Label smoothing with larger alpha, required for higher accuracy, substantially worsens calibration (ECE up to 0.67 on CIFAR10 at alpha 0.9).
Limitations
- Optimal alpha and V must be tuned per model and dataset via grid search.
- Robust distillation and adversarial training results rely on reweighting the robust regularization term beta to balance accuracy and robustness.
Experimental setup
Datasets: SVHN CIFAR10 CIFAR100 Tiny-ImageNet-200 ImageNet 300K Random Images
Benchmarks: SVHN CIFAR10 CIFAR100 Tiny-ImageNet-200 ImageNet Auto-Attack OOD detection mixture (Textures, Places365, iSUN, LSUN, SVHN)
Metrics: test accuracy ECE robust accuracy (PGD-20, CW, AA, AA+) clean accuracy AUC FPR-95
Baselines: one-hot label smoothing AT TRADES AWP RST OE SSL
Builds on: label smoothing knowledge distillation adversarial training TRADES AWP outlier exposure
Closest papers by meaning
- MaxSup: Overcoming Representation Collapse in Label Smoothing (NeurIPS 2025 · similarity 0.90)
- Robust Training under Label Noise by Over-parameterization (ICML 2022 · similarity 0.88)
- Does label smoothing mitigate label noise? (ICML 2020 · similarity 0.88)
- Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement (CVPR 2024 · similarity 0.88)
- Curriculum-Enhanced Residual Soft An-Isotropic Normalization for Over-Smoothness in Deep GNNs (AAAI 2024 · similarity 0.86)
- Identifying and Understanding Cross-Class Features in Adversarial Training (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.