Research map / Architectures and efficiency / RNNs and normalization
1-Lipschitz Layers Compared: Memory Speed and Certifiable Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically and theoretically compares seven 1-Lipschitz convolution parameterizations (AOL, BCOP, CPL, Cayley, LOT, SLL, SOC) on training/inference time, memory, accuracy and certified robust accuracy under fixed time budgets.
Problem
Existing 1-Lipschitz layer methods for certifiable robustness differ widely in training time, memory, and accuracy, making it hard to choose among them, especially under resource constraints.
Approach
A fair, resource-aware benchmark that fixes training wall-clock budgets (2h, 10h, 24h on one A100) rather than epochs, combined with a theoretical complexity and memory analysis of each layer's input and parameter transformations, and revised re-implementations released for comparison.
Key results
- SOC reaches the highest certified robust accuracy on two datasets (e.g., 66.3% at M size on CIFAR-10 at eps=36/255 for the M model).
- CPL ranks in the top-10 consistently across datasets and is recommended overall for its accuracy and low resource use; CPL is best on Imagenette.
- Cayley and LOT have much longer training times and memory costs; LOT-L does not fit in 80GB GPU memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOC instead of the other compared methods | 1-Lipschitz convolutional networks · CIFAR-10 | certified robust accuracy: 66.3 | Helped |
| Use SOC instead of the other compared methods | 1-Lipschitz convolutional networks · CIFAR-100 | certified robust accuracy: 34.9 | Helped |
| Use LOT instead of the other compared methods | 1-Lipschitz convolutional networks · Tiny ImageNet | certified robust accuracy: 21.9 | Helped |
| Use CPL instead of the other compared methods | 1-Lipschitz convolutional networks · Imagenette | accuracy | Helped |
| Use layers with a skip connection (CPL, SLL, or SOC) | 1-Lipschitz networks · evaluated datasets | certified robust accuracy | Helped |
| Use identity initialization (AOL or LOT) | 1-Lipschitz networks · evaluated datasets | certified robust accuracy | Helped |
| Use a pure contractive layer (AOL, CPL, or SLL) rather than an orthogonal layer | 1-Lipschitz networks · evaluated datasets | certified robust accuracy | No clear effect |
Surprising findings
- Slower methods (SOC, LOT) are among the best performers, contradicting the hypothesis that slower methods lose out under a fixed time budget.
- Pure contractive layers (AOL, CPL, SLL) do not show vanishing-gradient problems compared with orthogonal ones.
Limitations
- Only convolutional architectures with a fixed standard CNN backbone and MaxMin activation are evaluated.
- CPL is sensitive to weight initialization and showed numerical errors in some runs.
- Comparison is restricted to fixed training time budgets, so conclusions depend on the chosen hardware and budget.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet Imagenette
Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet Imagenette
Metrics: certified robust accuracy clean accuracy training time per epoch inference throughput training memory inference memory
Baselines: AOL BCOP CPL Cayley LOT SLL SOC Standard convolution
Builds on: Lipschitz bounded networks MaxMin activation AOL Cayley transform Convex Potential Layer
Closest papers by meaning
- A Recipe for Improved Certifiable Robustness (ICLR 2024 · similarity 0.91)
- Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds (NeurIPS 2021 · similarity 0.90)
- Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss (ICML 2025 · similarity 0.88)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.88)
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.87)
- Efficient Robust Conformal Prediction via Lipschitz-Bounded Networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.