Research map / Adversarial robustness and security / Certified and empirical robustness
Efficient local linearity regularization to overcome catastrophic overfitting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELLE, a cheap regularizer that enforces local linearity of the loss via a single random triplet (x_a, x_b, alpha) without double backpropagation, plus an adaptive variant (ELLE-A), to avoid catastrophic overfitting in single-step adversarial training.
Problem
Single-step adversarial training suffers catastrophic overfitting (CO) at large perturbation budgets or long schedules, while existing local-linearity regularizers avoid CO but cost about 3x runtime due to double backpropagation.
Approach
Regularize a local linear approximation error computed from one random triplet of points in the epsilon-ball (no input gradients, so no double backprop), with an adaptive lambda that increases only when this error spikes, linked theoretically to second-order directional derivatives (curvature).
Key results
- With PreActResNet18 on CIFAR10 at epsilon=12/255, N-FGSM+ELLE-A raises AutoAttack accuracy from 21.18% to 26.35%.
- ELLE takes 6.29 min of training versus 15.89 min for GradAlign and 5.06 min for FGSM on CIFAR10 (short schedule, eps=8/255).
- ELLE avoids catastrophic overfitting in all tested setups including the long schedule, where N-FGSM suffers CO for large epsilon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive ELLE to N-FGSM | N-FGSM · ImageNet, ε=8/255 | PGD-50-10 accuracy: 13.32% to 14.58% | Helped |
| Add adaptive ELLE to N-FGSM | N-FGSM · SVHN, largest ε (12/255) | AutoAttack accuracy: +5.17% | Helped |
| Add adaptive ELLE to N-FGSM | N-FGSM · CIFAR10, largest ε (26/255) | AutoAttack accuracy: +1.07% | Helped |
| Add adaptive ELLE to N-FGSM | N-FGSM · CIFAR100, largest ε (26/255) | AutoAttack accuracy: +3.20% | Helped |
| Adapt the local linearity regularization parameter during training | ELLE · CIFAR10, ε=8/255 | AutoAttack accuracy: 42.78 % v.s. 44.32 % | Helped |
| Add local linearity regularization to FGSM training | FGSM · CIFAR10, ε=8/255, Short schedule | training time: 5.06 min -> 6.29 min | Hurt |
| Add local linearity regularization to N-FGSM | N-FGSM · ImageNet, ε=8/255 | PGD-50-10 accuracy: 13.32 -> 14.58 | Helped |
| Remove adversarial attacks during training while retaining ELLE regularization (ablation) | ELLE · SVHN, ε=12/255 | PGD-20 accuracy: 32.17% -> 5.07% | Hurt |
| Use two triplet samples instead of one (ablation) | ELLE · CIFAR10, ε=8/255 | local linearity approximation error | No clear effect |
| Fix α to 0.5 instead of sampling it uniformly (ablation) | ELLE · CIFAR10, ε=8/255 | local linearity approximation error | No clear effect |
| Select α to maximize the local linearity approximation error (ablation) | ELLE · CIFAR10, ε=8/255 | local linearity approximation error | No clear effect |
| Increase GradAlign regularization to λ=8 (ablation) | GradAlign · CIFAR10, ε=26/255 | accuracy: 15.90 % v.s. 17.80 % | Hurt |
Surprising findings
- Regularizing local linearity without any adversarial attacks during training gives non-trivial robustness, though far below the full method (5.07% vs 32.17% PGD-20 at eps=12/255 on SVHN).
Limitations
- Requires forward passes on three extra points per training sample, increasing memory usage.
- Lambda must be tuned by grid search, and performance is sensitive to its value (too small leaves CO, too large degrades clean/robust accuracy).
- Large-epsilon robustness claims rest on a few datasets and architectures; robust overfitting appears in long schedules for the adaptive variant.
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN ImageNet
Benchmarks: CIFAR10 CIFAR100 SVHN ImageNet AutoAttack
Metrics: AutoAttack accuracy PGD-20 accuracy clean accuracy training runtime
Baselines: GradAlign CURE LLR SLAT Fast BAT N-FGSM GAT AT PGD-10
Builds on: FGSM adversarial training GradAlign LLR N-FGSM GAT
Closest papers by meaning
- Consistency Regularization for Adversarial Robustness (AAAI 2022 · similarity 0.90)
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.87)
- Layer-Aware Analysis of Catastrophic Overfitting: Revealing the Pseudo-Robust Shortcut Dependency (ICML 2024 · similarity 0.87)
- Boosting Adversarial Training via Fisher-Rao Norm-based Regularization (CVPR 2024 · similarity 0.86)
- Benign Overfitting in Adversarial Training of Neural Networks (ICML 2024 · similarity 0.86)
- Taxonomy Driven Fast Adversarial Training (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.