Research map / Adversarial robustness and security / Certified and empirical robustness
Lyapunov-Stable Deep Equilibrium Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LyaDEQ, a DEQ model whose fixed points are made Lyapunov-stable via an input-convex Lyapunov function and projected dynamics, plus an orthogonal FC layer to separate fixed points, improving adversarial robustness on MNIST, SVHN, CIFAR10/100.
Problem
DEQ models' adversarial robustness is largely unexplored, and the stability of their fixed points is poorly understood; existing certified methods give limited gains on complex image datasets.
Approach
Treats a DEQ layer as a nonlinear dynamical system and enforces Lyapunov stability of its fixed points using an input-convex neural network Lyapunov function with a projection onto a decrease-constrained dynamics set; an orthogonal FC layer after the stability module spreads apart stable equilibria of different classes.
Key results
- Under PGD attack at epsilon=8/255, LyaDEQ with orthogonal FC improves over the DEQ baseline by 5.12% on MNIST, 19.25% on SVHN, 32.48% on CIFAR10 and 16.45% on CIFAR100 (as stated in the text).
- On CIFAR10 under I-FGSM at epsilon=2/255, LyaDEQ combined with TRADES reaches 72.81% accuracy versus 48.48% for PGD-AT-combined variant reported in the same table, showing adversarial training stacks with LyaDEQ.
- At PGD epsilon=8/255 the LyaDEQ model is 7.38%, 7.50%, and 8.85% more accurate than ResNet56, VGG16, and WideResNet respectively, while being weaker at small epsilon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Lyapunov stability module to DEQ | DEQ · SVHN, PGD ε=8/255 | accuracy: 51.22 -> 70.47 | Helped |
| Add Lyapunov stability module to DEQ | DEQ · CIFAR10, PGD ε=8/255 | accuracy: 11.12 -> 43.96 | Helped |
| Add Lyapunov stability module to DEQ | DEQ · CIFAR100, PGD ε=8/255 | accuracy: 3.10 -> 19.55 | Helped |
| Add Lyapunov stability module to DEQ | DEQ · MNIST, PGD ε=8/255 | accuracy: 45.23 -> 50.35 | Helped |
| Use LyaDEQ instead of IBP-MonDEQ | DEQ · MNIST, PGD ε=8/255 | accuracy: 94.57 vs 50.35 | Hurt |
| Use LyaDEQ instead of IBP-MonDEQ | DEQ · SVHN, PGD ε=8/255 | accuracy: 52.04 -> 70.47 | Helped |
| Add TRADES adversarial training | LyaDEQ · CIFAR10, I-FGSM ε=2/255 | accuracy: +25.43% | Helped |
| Add robust dataset training | LyaDEQ · CIFAR10, I-FGSM ε=2/255 | accuracy: +4.58% | Helped |
| Add PGD adversarial training | LyaDEQ · CIFAR10, I-FGSM ε=2/255 | accuracy: +13.08% | Helped |
| Add orthogonal FC layer (ablation) | LyaDEQ · MNIST, PGD ε=8/255 | accuracy: 23.23 -> 50.35 | Helped |
| Add orthogonal FC layer (ablation) | LyaDEQ · SVHN, PGD ε=8/255 | accuracy: 65.60 -> 70.47 | Helped |
| Add orthogonal FC layer (ablation) | LyaDEQ · CIFAR10, PGD ε=8/255 | accuracy: 46.12 -> 43.96 | Hurt |
Surprising findings
- IBP-MonDEQ, a certified-training baseline, is far stronger on MNIST (about 95-96% under attack) but much weaker on SVHN and CIFAR, and LyaDEQ is weaker than conventional CNNs at small epsilon.
- Adding the orthogonal FC layer to the plain DEQ model does not significantly help and sometimes hurts accuracy.
Limitations
- Robustness at small perturbation radii (2/255, 4/255) is inferior to conventional CNNs.
- Performance on MNIST is far below the IBP-MonDEQ certified baseline.
- Clean accuracy of LyaDEQ on CIFAR100 slightly below baseline DEQ.
Experimental setup
Datasets: MNIST SVHN CIFAR10 CIFAR100
Benchmarks: MNIST SVHN CIFAR10 CIFAR100
Metrics: classification accuracy under adversarial attack
Baselines: DEQ IBP-MonDEQ ResNet56 VGG16 WideResNet
Builds on: deep equilibrium models IBP-MonDEQ input-convex neural networks Anderson acceleration TRADES
Closest papers by meaning
- A Closer Look at the Adversarial Robustness of Deep Equilibrium Models (NeurIPS 2022 · similarity 0.92)
- Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics (ICML 2023 · similarity 0.92)
- Positive Concave Deep Equilibrium Models (ICML 2024 · similarity 0.89)
- Separation and Bias of Deep Equilibrium Models on Expressivity and Learning Dynamics (NeurIPS 2024 · similarity 0.87)
- Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing (NeurIPS 2024 · similarity 0.85)
- Stability and Generalization of Adversarial Training for Shallow Neural Networks with Smooth Activation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.