Research map / Federated learning and privacy / Differential privacy
DP-SGD Without Clipping: The Lipschitz Neural Network Way
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Clipless DP-SGD, which replaces per-sample gradient clipping with analytically bounded per-layer sensitivities by training Lipschitz-constrained (GNP) networks, and releases the lip-dp TensorFlow package.
Problem
DP-SGD relies on per-sample gradient clipping, which biases gradient direction and is costly in memory and compute; tight sensitivity bounds for network layers are hard to obtain.
Approach
Link the Lipschitz constant with respect to inputs to the one with respect to parameters, so that Lipschitz-constrained layers (projected onto spectral-norm balls) give analytic per-layer gradient sensitivities computed by backpropagating scalar bounds, removing the need for clipping; GNP networks and loss-gradient clipping improve the signal-to-noise ratio.
Key results
- On CIFAR-10 under (20, 1e-5)-DP, certifiable accuracy at radius 0 reaches 47.4% with a Lipschitz network (tau=0.01 setting), with robustness certificates that vanilla DP-SGD cannot provide.
- Clipless DP-SGD's per-batch runtime and memory scale much better than vanilla DP-SGD and remain compatible with large batch sizes, per the speed benchmark on CIFAR-10 CNNs of 130K to 2M parameters.
- Tabular AUROC results on Adbench at epsilon=1 are reported as comparable to DP-SGD baselines (exact numbers in tables not extracted).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase softmax temperature from 0.01 to 0.22 (ablation) | Clipless DP-SGD · CIFAR-10, r=0/255 | certifiable accuracy: 47.4 -> 44.4 | Hurt |
| increase softmax temperature from 0.01 to 0.22 (ablation) | Clipless DP-SGD · CIFAR-10, r=1/255 | certifiable accuracy: 5.9 -> 39.1 | Helped |
| increase softmax temperature from 0.01 to 0.22 (ablation) | Clipless DP-SGD · CIFAR-10, r=2/255 | certifiable accuracy: 0.1 -> 34.2 | Helped |
| increase softmax temperature from 0.01 to 0.22 (ablation) | Clipless DP-SGD · CIFAR-10, r=4/255 | certifiable accuracy: 0.0 -> 24.9 | Helped |
| increase softmax temperature from 0.01 to 0.22 (ablation) | Clipless DP-SGD · CIFAR-10, r=8/255 | certifiable accuracy: 0.0 -> 11.5 | Helped |
| increase softmax temperature from 0.01 to 0.22 (ablation) | Clipless DP-SGD · CIFAR-10, r=16/255 | certifiable accuracy: 0.0 -> 0.8 | Helped |
| increase softmax temperature from 0.22 to 0.40 (ablation) | Clipless DP-SGD · CIFAR-10, r=0/255 | certifiable accuracy: 44.4 -> 41.6 | Hurt |
| increase softmax temperature from 0.22 to 0.40 (ablation) | Clipless DP-SGD · CIFAR-10, r=2/255 | certifiable accuracy: 34.2 -> 35.6 | Helped |
| increase softmax temperature from 0.22 to 0.40 (ablation) | Clipless DP-SGD · CIFAR-10, r=4/255 | certifiable accuracy: 24.9 -> 29.6 | Helped |
| increase softmax temperature from 0.22 to 0.40 (ablation) | Clipless DP-SGD · CIFAR-10, r=8/255 | certifiable accuracy: 11.5 -> 19.1 | Helped |
| increase softmax temperature from 0.22 to 0.40 (ablation) | Clipless DP-SGD · CIFAR-10, r=16/255 | certifiable accuracy: 0.8 -> 5.3 | Helped |
| increase softmax temperature from 0.40 to 0.74 (ablation) | Clipless DP-SGD · CIFAR-10, r=0/255 | certifiable accuracy: 41.6 -> 38.4 | Hurt |
Surprising findings
- Robust decisions and privacy are not necessarily antipodal; Lipschitz models provide certified robustness alongside DP guarantees.
- Clipping the loss gradient with respect to logits (a hybrid approach) can be seen as equivalent in direction to the Kantorovich-Rubinstein loss for binary classification.
Limitations
- Strict orthogonality constraints (GNP) are hard to enforce, especially for convolutions, and high-performing architectures differ from standard CNNs.
- Hyper-parameter search in baselines ignores the privacy loss from tuning, and the Lipschitz approach relies on projections rather than trivializations.
- Exact Lipschitz bound computation is NP-hard, so the propagated bounds can be loose.
Experimental setup
Datasets: MNIST CIFAR-10 Adbench tabular datasets
Benchmarks: MNIST CIFAR-10 Adbench
Metrics: validation accuracy AUROC certifiable accuracy median runtime per batch epsilon
Baselines: DP-SGD (Opacus) tf_privacy optax Opacus
Builds on: DP-SGD AutoLip Gaussian mechanism Renyi differential privacy accountant GNP networks
Closest papers by meaning
- Beyond Uniform Lipschitz Condition in Differentially Private Optimization (ICML 2023 · similarity 0.92)
- Direct Parameterization of Lipschitz-Bounded Deep Networks (ICML 2023 · similarity 0.90)
- Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks (NeurIPS 2019 · similarity 0.90)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.90)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.88)
- Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.