Research map / Federated learning and privacy / Differential privacy
Online Sensitivity Optimization in Differentially Private Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OSO-DPSGD, which treats the DP-SGD clipping threshold (and learning rate) as learnable parameters updated online by sign-based gradient steps, with a small extra privacy cost, and compares it to grid-searched fixed-threshold, fixed-quantile, and AdamWOSM baselines on MNIST, Fashion-MNIST and AG News.
Problem
Choosing the DP-SGD gradient clipping threshold is critical (bias vs. noise trade-off) but usually requires grid search, whose privacy cost is often ignored or large.
Approach
Derive a chain-rule gradient for the clipping threshold C_t using a masked previous-iteration gradient (unit-sensitivity query), then use a sign-based exponential update for C_t and the learning rate, privatized by splitting the noise budget between two parallel Gaussian queries.
Key results
- On MNIST at eps=9 (k=9 grid), OSO-DPSGD reaches 93.62% accuracy vs 90.72% for FixedThreshold and 91.09% for FixedQuantile.
- On AG News, AdamWOSM performs best and OSO-DPSGD is marginally below, which the authors consider comparable within a standard deviation.
- FixedQuantile diverges for all runs at eps=1 on Fashion MNIST, while OSO-DPSGD converges there (MSE 2.57 x10^-2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online clipping-threshold optimization instead of AdamWOSM | DP-SGD with the MNIST CNN · MNIST, ε=3 and ε=5 | accuracy: 90.69 vs 90.62; 91.62 vs 91.60 | No clear effect |
| Use online clipping-threshold optimization instead of AdamWOSM | DP-SGD with the MNIST CNN · MNIST, ε=7 and ε=9 | accuracy: 92.19 vs 92.94; 92.53 vs 93.62 | Helped |
| Use online clipping-threshold optimization instead of FixedQuantile | DP-SGD with the FashionMNIST autoencoder · FashionMNIST, ε=2–4 | MSE × 10²: 0.94 vs 2.17; 0.74 vs 1.26; 0.76 vs 1.05 | Helped |
| Use online clipping-threshold optimization instead of AdamWOSM | DP-SGD with the AG News classifier · AG News, ε=3–9 | accuracy: 85.70 vs 85.82; 86.12 vs 86.13; 86.34 vs 86.33; 86.43 vs 86.43 | No clear effect |
| Use online clipping-threshold optimization instead of FixedQuantile | DP-SGD with the FashionMNIST autoencoder · FashionMNIST, ε=1 | MSE × 10² | Helped |
Surprising findings
- The adaptive FixedQuantile strategy did not outperform a fixed clipping threshold in centralized learning, contrary to federated results in prior work.
- AdamWOSM is penalized by its suggested initial learning rate, which combined with small optimal clipping thresholds slows convergence.
Limitations
- Adam extension is not derived; analysis limited to vanilla SGD and momentum-trivial cases.
- Privacy noise split between queries relies on a heuristic (nu_q approx 7.124 nu) rather than a tight analysis.
- Gains are mostly comparable rather than clearly superior; OSO-DPSGD is below AdamWOSM on AG News.
Experimental setup
Datasets: MNIST FashionMNIST AG News
Benchmarks: MNIST FashionMNIST AG News
Metrics: accuracy mean squared error standard deviation over seeds
Baselines: FixedThreshold FixedQuantile AdamWOSM NoDP
Builds on: DP-SGD online learning rate optimization (hypergradient descent) adaptive clipping (quantile-based) AdamWOSM
Closest papers by meaning
- The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection (AAAI 2022 · similarity 0.92)
- On Optimal Hyperparameters for Differentially Private Deep Transfer Learning (ICLR 2026 · similarity 0.91)
- Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.90)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.90)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.89)
- A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.