Research map / Federated learning and privacy / Differential privacy
Revisiting Differentially Private ReLU Regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-GLMtron (single-pass private GLM-tron with adaptive clipping) and DP-TAGLMtron (tree-aggregation variant) for DP ReLU regression in overparameterized regimes, with excess-risk bounds that can be dimension-independent under eigenvalue decay.
Problem
Differentially private ReLU regression is non-convex and existing excess-risk bounds assume bounded ||x||_2 <= 1 or become trivial when dimension d exceeds sample size N, i.e., in the overparameterized regime.
Approach
Adapts the GLM-tron perceptron-style update (dropping the ReLU derivative) with adaptive clipping via a private residual estimator and Gaussian noise, using shuffling amplification; a tree-aggregation variant removes the need for small epsilon. Utility bounds are expressed via effective dimensions defined by covariance eigenvalues rather than d.
Key results
- DP-GLMtron achieves excess population risk upper bound of O~(D_eff/N + D^pri_eff/(N^2 eps^2)), with a matching lower bound up to log factors.
- DP-TAGLMtron incurs only an extra O(log N) factor over DP-GLMtron while removing the need for eps = O~(1/sqrt(N)).
- On synthetic and MNIST experiments, DP-GLMtron and DP-TAGLMtron outperform DP-SGD and DP-FTRL in excess risk, especially as sample size grows; faster eigenvalue decay gives lower excess risk.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DP-GLMtron instead of DP-SGD or DP-FTRL | ReLU regression · Synthetic Bernoulli, λ_i ∝ i^-2 and λ_i ∝ i^-3 | excess risk | Helped |
| use DP-TAGLMtron instead of DP-SGD or DP-FTRL | ReLU regression · Synthetic Bernoulli, λ_i ∝ i^-2 and λ_i ∝ i^-3 | excess risk | Helped |
| use faster eigenvalue decay, λ_i ∝ i^-3 rather than λ_i ∝ i^-2 | ReLU regression algorithms · Synthetic Bernoulli | excess risk | Helped |
| increase the privacy budget from ε = 0.05 to ε = 0.5 | DP-GLMtron and DP-TAGLMtron · Synthetic Bernoulli ReLU regression | excess risk | Helped |
| use DP-TAGLMtron instead of the baselines | ReLU regression · MNIST | excess risk | Helped |
| use DP-TAGLMtron instead of DP-SGD or DP-FTRL | ReLU regression · MNIST | excess risk | Helped |
Surprising findings
- Utility bound can be independent of dimension d even when d >> N under polynomial or exponential eigenvalue decay.
Limitations
- DP-GLMtron's upper bound requires small privacy budget eps = O~(1/sqrt(N)) due to shuffling amplification.
- Analysis is restricted to ReLU regression and does not clearly generalize to other non-convex problems.
- Analysis assumes well-specified model and symmetry/fourth-moment assumptions on data.
Experimental setup
Datasets: MNIST synthetic Bernoulli data with eigenvalue decay
Benchmarks: MNIST synthetic Bernoulli data
Metrics: excess risk
Baselines: DP-SGD DP-FTRL DP-PGD
Builds on: GLM-tron DP-SGD DP-FTRL privacy amplification by shuffling
Closest papers by meaning
- 𝑓-Differential Privacy Filters: Validity and Approximate Solutions (NeurIPS 2026 · similarity 0.89)
- Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacy (NeurIPS 2025 · similarity 0.89)
- The Limits of Differential Privacy in Online Learning (NeurIPS 2024 · similarity 0.88)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.88)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.88)
- Noise-Aware Differentially Private Regression via Meta-Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.