Research map / Learning theory and algorithms / Prediction and decision losses
Monotone, Bi-Lipschitz, and Polyak-Łojasiewicz Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiLipNet, a bi-Lipschitz invertible network built from IQC-certified strongly monotone feed-through residual layers and orthogonal layers, and PLNet, a scalar network (BiLipNet composed with quadratic potential) satisfying the Polyak-Łojasiewicz condition with efficiently computable unique global minimum.
Problem
Learning invertible or scalar networks with certified bi-Lipschitz bounds and, for surrogate losses, a unique and efficiently computable global minimum, where spectral-normalization-based bounds are very conservative.
Approach
A feed-through residual layer mu*x + H(x) whose monotonicity and Lipschitzness are certified via incremental quadratic constraints and a Cayley-transform direct parameterization; composed with orthogonal layers into BiLipNet; the inverse is computed via Davis-Yin three-operator splitting, and PLNet = 1/2||G(x)||^2 inherits the PL condition from the inverse-Lipschitz bound.
Key results
- On CIFAR-10 with c=0.95, BiLipNet reaches 86.2% clean accuracy versus 76.7% for SNGP under similar bi-Lipschitz bounds.
- On 20-D Rosenbrock, BiLipNet achieves far better train/test fits than i-ResNet and i-DenseNet across distortion values, with training error near 1e-8 at small distortion.
- The DYS three-operator solver converges much faster than forward step method and ADAM for computing the PLNet global minimum.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SNGP spectral-normalized components with BiLipNet components | SNGP · CIFAR-10 clean and corrupted | accuracy: 76.7 $\pm$ 0.629 -> 86.7 $\pm$ 0.129; 58.7 $\pm$ 1.000 -> 72.8 $\pm$ 0.592 | Helped |
| Replace SNGP spectral-normalized components with BiLipNet components | SNGP · CIFAR-10 clean and corrupted | ECE: 0.057 $\pm$ 0.007 -> 0.015 $\pm$ 0.005; 0.079 $\pm$ 0.006 -> 0.047 $\pm$ 0.009 | Helped |
| Replace SNGP spectral-normalized components with BiLipNet components | SNGP · CIFAR-10 clean and corrupted | NLL: 0.682 $\pm$ 0.015 -> 0.400 $\pm$ 0.006; 1.199 $\pm$ 0.041 -> 0.830 $\pm$ 0.024 | Helped |
| Replace SNGP spectral-normalized components with BiLipNet components | SNGP · CIFAR-100 clean and corrupted | accuracy: 36.9 $\pm$ 1.656 -> 55.2 $\pm$ 0.426; 25.5 $\pm$ 1.406 -> 39.2 $\pm$ 0.495 | Helped |
| Replace SNGP spectral-normalized components with BiLipNet components | SNGP · CIFAR-100 clean and corrupted | NLL: 2.493 $\pm$ 0.068 -> 1.887 $\pm$ 0.021; 3.073 $\pm$ 0.069 -> 2.576 $\pm$ 0.022 | Helped |
| Replace SNGP spectral-normalized components with BiLipNet components | SNGP · CIFAR-100 clean and corrupted | ECE: 0.131 $\pm$ 0.010 -> 0.225 $\pm$ 0.004; 0.068 $\pm$ 0.005 -> 0.137 $\pm$ 0.005 | Hurt |
| Vary distortion parameter c across 0.95, 0.65, and 0.35 (ablation) | BiLipNet · CIFAR-10 and CIFAR-100 | accuracy | Mixed |
Surprising findings
- Spectral-normalized SNGP with small distortion (c=0.35, tau=18.6) outperforms SNGP with much larger distortion, showing BiLipNet is more expressive per distortion bound.
- Batch normalization can break certified bi-Lipschitz bounds, so it was removed from the SNGP baseline.
Limitations
- Inverse computation requires an iterative solver, unlike the explicit forward pass
- Gradient descent on PLNet may fail because gradients are not globally Lipschitz
- Convergence of DYS requires step size restricted by model distortion
Experimental setup
Datasets: CIFAR-10 CIFAR-100 two-moon Rosenbrock function (synthetic) Rosenbrock+Sine (synthetic)
Benchmarks: CIFAR-10 CIFAR-100 two-moon OOD toy 2D Rosenbrock 20-D Rosenbrock
Metrics: accuracy expected calibration error negative log likelihood train/test error vs distortion convergence rate of solver
Baselines: SNGP i-ResNet i-DenseNet ICNN MLP forward step method ADAM
Builds on: i-ResNet spectral normalization Davis-Yin splitting input convex neural network SNGP
Closest papers by meaning
- A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness (NeurIPS 2024 · similarity 0.90)
- Approximation theory for 1-Lipschitz ResNets (NeurIPS 2025 · similarity 0.88)
- Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss (ICML 2025 · similarity 0.86)
- Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations (ICLR 2024 · similarity 0.85)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.85)
- Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.