Research map / Optimization / Implicit bias and generalization
Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Convexifies a two-layer ReLU network via a stochastic dual (random hyperplane) formulation with squared regularization, enabling hidden-state DP analysis of NoisyCGD that matches DP-SGD on 2-layer ReLU nets for MNIST/CIFAR-10.
Problem
Hidden-state differential privacy analyses only apply to convex problems, and the best convex models (logistic regression) underperform neural networks, so private training of multi-layer networks under the hidden-state threat model lacks accurate guarantees.
Approach
Approximate the convex dual of the two-layer ReLU problem with a stochastic, unconstrained, strongly convex GLM (random hyperplane activation patterns, squared L2 regularization), so that privacy amplification by iteration (GDP bounds) applies to noisy cyclic mini-batch GD with disjoint batches.
Key results
- On MNIST at epsilon=4.76, NoisyCGD on the convex approximation reaches 94.4% vs 94.3% for DP-SGD on a ReLU network (5 restarts).
- On CIFAR-10 at epsilon=4.76, NoisyCGD convex approximation reaches 45.4% vs 47.0% for DP-SGD + ReLU.
- The convex models significantly outperform DP logistic regression, the previous best model for hidden-state DP analysis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the convex approximation instead of the ReLU network with DP-SGD | DP-SGD image classification · MNIST test, ε=1.33 and ε=4.76 | accuracy: 91.7 ± 0.1 -> 93.1 ± 0.1; 94.3 ± 0.1 -> 94.9 ± 0.1 | Helped |
| Use the convex approximation instead of the ReLU network with DP-SGD | DP-SGD image classification · CIFAR-10 test, ε=1.33 and ε=4.76 | accuracy: 42.5 ± 0.1 -> 41.5 ± 0.2; 47.0 ± 0.2 -> 45.5 ± 0.2 | Hurt |
| Use NoisyCGD with the convex approximation instead of DP-SGD with the ReLU network | Private image classification · MNIST test, ε=1.33 and ε=4.76 | accuracy: 91.7 ± 0.1 -> 92.4 ± 0.2; 94.3 ± 0.1 -> 94.4 ± 0.1 | Mixed |
| Use NoisyCGD with the convex approximation instead of DP-SGD with the ReLU network | Private image classification · CIFAR-10 test, ε=1.33 and ε=4.76 | accuracy: 42.5 ± 0.1 -> 41.0 ± 0.2; 47.0 ± 0.2 -> 45.4 ± 0.3 | Hurt |
| Use an iterative method instead of SSP with the same convex-approximation features | Private image classification · MNIST and CIFAR-10 | accuracy | Helped |
| Increase the number of random hyperplanes (ablation) | Stochastic convex approximation · MNIST classification, non-private SGD | test accuracy | Helped |
| Use feature normalization to set each feature vector's ℓ2 norm to 5.0 (ablation) | NoisyCGD and DP-SGD models · MNIST and CIFAR-10 | accuracy | Mixed |
Surprising findings
- Sufficient statistics perturbation on random or RFF features is clearly worse than iterative methods, even though it is a one-step solution.
Limitations
- Utility guarantee for the random data model requires n/d >= 1 and bounded gradients, and the analysis assumes a constant ratio c.
- Only shallow (2-layer) networks are addressed; extension to deeper networks is left open.
- Convex approximation with random hyperplanes is data-dependent in its constraints for the unrelaxed problem; the final model uses an unconstrained approximation requiring a learning rate condition eta < 2/(beta+lambda) not strictly enforced.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR-10
Benchmarks: MNIST FashionMNIST CIFAR-10
Metrics: test accuracy epsilon (DP)
Baselines: DP-SGD + ReLU DP-SGD + logistic regression Sufficient Statistics Perturbation Random Fourier Features
Builds on: DP-SGD privacy amplification by iteration convex duality of two-layer ReLU networks Gaussian differential privacy sufficient statistics perturbation
Closest papers by meaning
- Revisiting Differentially Private ReLU Regression (NeurIPS 2024 · similarity 0.89)
- Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (ICLR 2025 · similarity 0.88)
- Improved Rates of Differentially Private Nonconvex-Strongly-Concave Minimax Optimization (AAAI 2025 · similarity 0.88)
- Rethinking DP-SGD in Discrete Domain: Exploring Logistic Distribution in the Realm of signSGD (ICML 2024 · similarity 0.87)
- Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness (ICLR 2025 · similarity 0.87)
- Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.