Research map / Adversarial robustness and security / Certified and empirical robustness
Low Curvature Activations Reduce Overfitting in Adversarial Training
Vasu Singla, Sahil Singla, Soheil Feizi, David Jacobs
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SiLU instead of LiSHT (ablation) | adversarially trained ResNet-18 · CIFAR-10 | robust generalization gap: 45.06 -> 7.24 | Improved |
| use SiLU instead of ReLU | adversarially trained ResNet-18 · CIFAR-100 | robust test accuracy (final): 18.81 -> 24.29 | Improved |
| use SiLU instead of ReLU | adversarially trained ResNet-18 · CIFAR-10 | robust generalization gap: 33.21 -> 7.24 | Improved |
| use SiLU instead of LiSHT (ablation) | adversarially trained ResNet-18 · CIFAR-10 | robust overfitting gap (best vs final): 3.1% -> 0.5% | Improved |
| use LiSHT (smooth, high curvature) instead of ReLU | adversarially trained ResNet-18 · CIFAR-10 | robust test accuracy (final): 49.25 -> 47.21 | Hurt |
| lower PSwish beta from 10 to 0.5 (lower curvature) (ablation) | PSwish ResNet-18 · CIFAR-10 | robust generalization gap: 38.57 -> 1.76 | Improved |
| lower PSwish beta from 10 to 0.5 (lower curvature) (ablation) | PSwish ResNet-18 · CIFAR-10 | robust test accuracy (final): 50.63 -> 45.24 | Hurt |
| lower PSwish beta from 10 to 0.5 (lower curvature) (ablation) | PSwish ResNet-18 · CIFAR-10 | standard generalization gap: 16.13 -> 1.82 | Improved |
| raise LeakyReLU slope k from -0.2 to 0.5 (lower approximate curvature) (ablation) | LeakyReLU ResNet-18 · CIFAR-10 | robust generalization gap: 37.73 -> 4.03 | Improved |
| raise LeakyReLU slope k from -0.2 to 0.5 (lower approximate curvature) (ablation) | LeakyReLU ResNet-18 · CIFAR-10 | standard generalization gap: 16.46 -> 3.43 | Improved |
| use PSwish beta=10 instead of ReLU with early stopping | adversarially trained ResNet-18 · CIFAR-10 | robust and standard accuracy: +0.33% robust, +1.24% standard | Improved |
| use SiLU instead of ReLU in wide ResNets | adversarially trained WideResNet · width sweep | robust test error vs width: width-factor 4 equals ReLU width-factor 15 | Improved |
Closest papers by meaning
- Robustness via Curvature Regularization, and Vice Versa (CVPR 2019 · similarity 0.89)
- Overfitting in adversarially robust deep learning (ICML 2020 · similarity 0.88)
- Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality (NeurIPS 2020 · similarity 0.88)
- Efficient local linearity regularization to overcome catastrophic overfitting (ICLR 2024 · similarity 0.87)
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.86)
- Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
