Research map / Federated learning and privacy / Differential privacy
Proactive DP: A Multiple Target Optimization Framework for DP-SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes "proactive DP", a closed-form refinement of the moment accountant for DP-SGD that lets practitioners pick sigma, epsilon, and rounds T a priori for a target privacy budget and utility, supported by a utility graph and DP calculator.
Problem
Existing DP accountants only track privacy spent after training, so DP-SGD parameters cannot be chosen a priori to jointly satisfy a privacy budget, a test-accuracy goal, and a communication budget.
Approach
A refined moment-accountant analysis yields a closed-form relation sigma = sqrt(2(eps+ln(1/delta))/eps) with a near-tight lower bound on the number of rounds T (about 2k^2/eps), and a utility graph (accuracy vs. sigma, C without private training) plus DP calculator are used to select parameters before training.
Key results
- Strongly convex: epsilon=0.04945 with sigma=19.2 achieves 86% test accuracy versus 93% without DP; epsilon=0.1 achieves 91%.
- Non-convex LeNet on MNIST: epsilon=0.15 reaches about 93% test accuracy versus 98% without DP.
- The paper claims DP-SGD is (eps,delta)-DP with sigma=sqrt(2(eps+ln(1/delta))/eps) when T is at least about 2k^2/eps, and that T cannot be reduced by more than a factor of about 4 while keeping the guarantee.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase ε from 0.04945 to 0.1 | Asynchronous DP-SGD · Strongly convex objective | test accuracy: 86% -> 91% | Helped |
| use ε=0.04945 with differential privacy instead of no-DP training | Asynchronous DP-SGD · Strongly convex objective | test accuracy: 93% -> 86% | Hurt |
| use ε=0.15 with differential privacy instead of no-DP training | Asynchronous DP-SGD · Non-convex LeNet image classification | test accuracy: 98% -> 93% | Hurt |
| increase the constant sample size to s=26 | Asynchronous DP-SGD · Strongly convex and plain convex objectives | test accuracy | Helped |
Surprising findings
- The paper argues that the number of rounds T should be chosen as small as possible (T near the lower bound), which contradicts the usual intuition of many small steps.
Limitations
- The utility graph is a heuristic based on perturbing a non-private model and does not guarantee the accuracy of actual DP-SGD training.
- Main theorem requires epsilon < 0.5 and delta <= 1/N unless the more general, less intuitive conditions are used.
- Experiments are small-scale (LIBSVM, MNIST, LeNet) with 5 simulated clients.
Experimental setup
Datasets: MNIST LIBSVM
Benchmarks: MNIST LIBSVM
Metrics: test accuracy epsilon
Baselines: Abadi et al. DP-SGD non-private training
Builds on: DP-SGD moment accountant f-DP Hogwild!
Closest papers by meaning
- The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD (ICLR 2025 · similarity 0.87)
- Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (ICLR 2025 · similarity 0.87)
- 𝑓-Differential Privacy Filters: Validity and Approximate Solutions (NeurIPS 2026 · similarity 0.87)
- How Private are DP-SGD Implementations? (ICML 2024 · similarity 0.87)
- Scalable DP-SGD: Shuffling vs. Poisson Subsampling (NeurIPS 2024 · similarity 0.87)
- Provable Privacy with Non-Private Pre-Processing (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.