Research map / Federated learning and privacy / Differential privacy
Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits DP-SGD in the hidden state threat model by having adversaries craft offline gradient sequences (not canaries), showing tight audits when inserted every step and partial tightness/amplification evidence otherwise.
Problem
Empirical privacy auditing of DP-SGD under the hidden state threat model (only final model released) shows a large gap between lower bounds and accounting upper bounds, and it is unclear whether the gap is due to weak adversaries or loose upper bounds.
Approach
Instead of crafting a canary data point, the adversary crafts an offline gradient sequence (biased in a single coordinate of magnitude C) inserted into DP-SGD, and for periodic insertion additionally designs a 1D non-convex loss landscape to maximally bias subsequent genuine gradients.
Key results
- With crafted gradient inserted at every step (k=1), the random-biased-dimension adversary matches the accountant's upper bound on CIFAR10 ConvNet/ResNet; on the 68-parameter FCNN the simulated-dimension adversary recovers nearly tight results.
- For periodicity k=5 and k=25 the adversary outperforms the loss-based canary baseline by a large margin but remains below the upper bound, with a gap that grows as k increases.
- With the loss-landscape adversary over T=25 steps, auditing is tight for large batch size relative to noise variance and shows privacy amplification (ratio below 1) for small batch sizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a crafted gradient instead of a loss-based canary adversary | DP-SGD hidden state auditing on CIFAR10 · CIFAR10, k=1 | privacy auditing lower bound | Helped |
| Use GC-S to select the least-updated dimension instead of random dimension selection | Gradient-crafting adversary on the Housing FCNN · Housing, k=1 | privacy auditing lower bound | Helped |
| Insert crafted gradients at every optimization step | DP-SGD in the hidden state threat model · Hidden state, k=1 | privacy auditing lower bound | Helped |
| Increase insertion periodicity beyond every step | Gradient-crafting adversary on CIFAR10 models · CIFAR10, k>1 | privacy auditing lower bound | Hurt |
| Use GC-R rather than the loss-based adversary | DP-SGD hidden state auditing · CIFAR10, k>1 | privacy auditing lower bound | Helped |
| Craft the loss landscape as well as the gradient | DP-SGD hidden state auditing · Large batch relative to noise variance | privacy auditing lower bound | Helped |
| Use a small batch relative to noise variance | DP-SGD hidden state model · Non-convex setting, small batch | privacy loss | Mixed |
| Use noiseless rather than noisy training simulation to select the biased dimension (ablation) | GC-S on Housing · Housing | privacy auditing lower bound | Helped |
| Rank dimensions by accumulated per-step updates rather than final-model distance (ablation) | GC-S on Housing · Housing | privacy auditing lower bound | Helped |
| Replace single-dimension gradient crafting with a random unit-sphere gradient and cosine score (ablation) | Hidden state auditing · CIFAR10 | privacy auditing lower bound | Hurt |
| Make the initialization unknown using Xavier initialization (ablation) | GC-R auditing · CIFAR10 | privacy auditing lower bound | No clear effect |
| Fine-tune a pretrained AlexNet rather than train a CNN from scratch (ablation) | GC-R auditing · CIFAR10, k=1 | privacy auditing lower bound | No clear effect |
Surprising findings
- Hiding intermediate models does not amplify privacy when the crafted gradient is inserted at every step, so the simplest random biased-dimension adversary is already tight for over-parameterized models.
- Privacy loss for the non-convex loss-landscape adversary converges to a positive constant rather than to zero, unlike the convex case.
Limitations
- For periodic insertion (k>1) the auditing lower bound remains loose relative to the accountant's upper bound.
- The loss-landscape adversary assumes an adversarially chosen non-convex loss that may not be realizable in standard architectures.
- Gaussian DP auditing approximates the trade-off function with a Gaussian one, which can underestimate privacy in some mechanisms.
Experimental setup
Datasets: CIFAR10 Housing
Benchmarks: CIFAR10 Housing
Metrics: empirical epsilon theoretical epsilon amplification rate
Baselines: loss-based canary adversary gradient cosine-similarity adversary
Builds on: DP-SGD Gaussian differential privacy privacy amplification by iteration gradient-crafting auditing (Nasr et al.)
Closest papers by meaning
- Nearly Tight Black-Box Auditing of Differentially Private Machine Learning (NeurIPS 2024 · similarity 0.92)
- The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD (ICLR 2025 · similarity 0.91)
- Auditing Private Prediction (ICML 2024 · similarity 0.89)
- Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacy (NeurIPS 2025 · similarity 0.88)
- A Unified Framework for Adversary-Aware Differential Privacy Bounds (NeurIPS 2026 · similarity 0.88)
- One-shot Empirical Privacy Estimation for Federated Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.