Research map / Learning theory and algorithms / Prediction and decision losses
A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a bi-Lipschitz neural network (BLNN) that directly parameterizes the overall bi-Lipschitzness via convex neural networks (ICNN) and a Legendre-Fenchel transform, and evaluates tightness, flexibility, annealing, uncertainty estimation, and monotone regression.
Problem
Controlling the sensitivity of neural networks by enforcing overall (not layer-wise) bi-Lipschitz bounds, which prior layer-wise Lipschitz methods only loosely achieve.
Approach
Build the network as the gradient of a strongly convex ICNN plus a quadratic term, with the inverse obtained via Legendre-Fenchel transformation, so that the derivative is guaranteed to lie between alpha and alpha+beta in Lipschitz sense; this gives tight, global bi-Lipschitz bounds instead of layer-wise products.
Key results
- On the Lipschitz tightness test with L=10, the proposed method reaches 99.4% of the bound versus 58.9% for BiLipNet and 68.9% for SN.
- In OOD detection with full-size data, DUQ+BLNN improves AUROC on NotMNIST from 0.900 to 0.964 over DUQ.
- On monotone benchmarks, the method achieves the best accuracy on COMPAS (69.4), LoanDefaulter (65.5), HeartDisease (90.2) and the best MSE on AutoMPG (7.13).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use direct bi-Lipschitz parameterization instead of layer-wise Lipschitz constraints | Lipschitz-constrained function fitting · Lipschitz-bound tightness, L=50 | bound tightness: 32.5\%$\pm$5.8 -> 99.9\%$\pm$0.1 | Helped |
| add BLNN bi-Lipschitz constraint to DUQ | DUQ · Downsampled FashionMNIST vs NotMNIST | AUROC: 0.852$\pm$.010 -> 0.960$\pm$.009 | Helped |
| add BLNN bi-Lipschitz constraint to DUQ | DUQ · Full-size FashionMNIST vs NotMNIST | AUROC: 0.900$\pm$.024 -> 0.964$\pm$.005 | Helped |
| add BLNN bi-Lipschitz constraint to DUQ | DUQ · Full-size FashionMNIST vs MNIST | AUROC: 0.862$\pm$.035 -> 0.896$\pm$.008 | Helped |
| use direct bi-Lipschitz constraints rather than layer-wise constraints | Lipschitz-constrained regression · Fitting y=50x over varying L | loss | Helped |
| use direct bi-Lipschitz parameterization rather than layer-wise methods | Function fitting with overestimated Lipschitz bound · Linear functions y=50x and y=x, specified L=50 | fit quality | Helped |
| increase the specified Lipschitz bound to 1000 | Lipschitz-constrained function fitting · Fitting y=x with L=1000 | generalization performance | Helped |
| use elastic Lipschitz-bound annealing | Ours · Fitting y=exp(x) | annealing speed | Helped |
| set α=5.0 (ablation) | DUQ+BLNN · Two Moons | accuracy | Hurt |
| set β=0.1 (ablation) | DUQ+BLNN · Two Moons | accuracy: 0.716$\pm$.111 | Hurt |
| choose β large enough while varying α (ablation) | DUQ+BLNN · Two Moons | accuracy | No clear effect |
| use a tighter β setting (ablation) | DUQ+BLNN · Two Moons | accuracy | Helped |
Surprising findings
- With default initialization, the estimated Lipschitz constant concentrates at the theoretical upper bound, indicating a strong initialization bias.
- Layer-wise methods such as SN and AOL fail to fit a linear function with slope 50 even when the bound is set to 50, while the proposed method fits it easily.
Limitations
- Initialization of the ICNN weights leads to biased bi-Lipschitz constants; adequate initialization is left open.
- Inverse computation via Legendre-Fenchel transform requires an iterative optimization (gradient descent up to 1000 iterations) per forward pass.
- On BlogFeedBack the method is slightly worse than SMNN in RMSE (0.157 vs 0.150).
Experimental setup
Datasets: two moons FashionMNIST MNIST NotMNIST COMPAS BlogFeedBack LoanDefaulter HeartDisease AutoMPG
Benchmarks: two moons FashionMNIST vs MNIST FashionMNIST vs NotMNIST COMPAS BlogFeedBack LoanDefaulter HeartDisease AutoMPG
Metrics: empirical Lipschitz tightness (%) loss AUROC accuracy RMSE MSE
Baselines: spectral normalization AOL Orthogonal SLL Sandwich LMN BiLipNet DUQ Deep Ensembles SMNN
Builds on: input convex neural networks spectral normalization DUQ LMN
Closest papers by meaning
- Some Fundamental Aspects about Lipschitz Continuity of Neural Networks (ICLR 2024 · similarity 0.90)
- Monotone, Bi-Lipschitz, and Polyak-Łojasiewicz Networks (ICML 2024 · similarity 0.90)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.88)
- Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss (ICML 2025 · similarity 0.87)
- Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations (ICLR 2024 · similarity 0.87)
- DP-SGD Without Clipping: The Lipschitz Neural Network Way (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.