Research map / Adversarial robustness and security / Certified and empirical robustness

Learning Better Certified Models from Empirically-Robust Teachers

NeurIPS 2026 · Adversarial robustness and security › Certified and empirical robustness · approach: Robustness-accuracy trade-off analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CC-Dist, which distills features from an adversarially-trained teacher into a certifiably-robust ReLU student trained with convex-combination IBP loss, improving standard and certified accuracy over CC-IBP on CIFAR-10, TinyImageNet and ImageNet64.

Problem

Certifiably robust models trained with expressive certified losses still have substantially lower standard accuracy and robustness-accuracy trade-offs than empirically robust models, leaving a gap between certified and empirical robustness.

Approach

A feature-space distillation loss from an adversarially-trained teacher whose worst-case upper and lower bounds are computed via IBP over convex combinations of adversarial student features and IBP bounds, with the same alpha as the CC-IBP expressive loss so the distillation is tightly coupled with the certified loss.

Key results

• On CIFAR-10 eps=2/255, CC-Dist reaches 81.55% standard and 64.60% certified accuracy vs 79.51%/63.50% for CC-IBP in the same codebase.
• On TinyImageNet eps=1/255, CC-Dist reaches 43.78% standard and 27.88% certified accuracy, improving on prior literature results.
• On ImageNet64 eps=1/255, CC-Dist reaches 28.17% standard and 13.96% certified accuracy, a new ReLU state of the art in the paper's comparison; on CIFAR-10 eps=8/255 SortNet keeps the best certified accuracy.

Surprising findings

• Adversarially-trained teachers have very low certified accuracy (e.g., 0-17%), yet their students significantly surpass them in verifiability.
• Decreasing alpha for CC-IBP on TinyImageNet and ImageNet64 improved over original literature settings.

Limitations

• Teachers use the same architecture as the student; larger-capacity teachers not explored.
• Hyperparameters were tuned directly on the evaluation sets, as in prior work.
• Certified accuracy of teachers evaluated only on first 500 test images due to verification cost; no gains reported for 1-Lipschitz SortNet on CIFAR-10 eps=8/255.

Experimental setup

Datasets: CIFAR-10 TinyImageNet ImageNet64

Benchmarks: CIFAR-10 TinyImageNet ImageNet64

Metrics: standard accuracy certified accuracy AutoAttack accuracy

Baselines: CC-IBP MTL-IBP STAPS SABR SortNet IBP-R IBP CROWN-IBP Exp-IBP

Builds on: CC-IBP expressive losses auto_LiRPA OVAL PGD adversarial training IBP regularization

Closest papers by meaning

• On the Scalability of Certified Adversarial Robustness with Generated Data (NeurIPS 2024 · similarity 0.90)
• Robust Distillation via Untargeted and Targeted Intermediate Adversarial Samples (CVPR 2024 · similarity 0.90)
• A Recipe for Improved Certifiable Robustness (ICLR 2024 · similarity 0.90)
• Et Tu Certifications: Robustness Certificates Yield Better Adversarial Examples (ICML 2024 · similarity 0.90)
• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.89)
• Towards Adversarially Robust Dataset Distillation by Curvature Regularization (AAAI 2025 · similarity 0.89)