Research map / Federated learning and privacy / Differential privacy
Nearly Tight Black-Box Auditing of Differentially Private Machine Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a black-box DP-SGD auditing procedure that crafts worst-case (non-privately pre-trained) initial parameters, yielding substantially tighter empirical epsilon estimates than prior average-case-initialization black-box audits.
Problem
Black-box auditing of DP-SGD has produced only loose empirical privacy lower bounds; it was an open question whether tight audits are possible when the adversary only sees the final model and can insert only an input canary.
Approach
Since DP-SGD's privacy guarantee holds for arbitrary fixed initial parameters, the auditor sets theta_0 to parameters obtained by non-private pre-training that minimize gradients of normal samples, so the target sample's gradient becomes much more distinguishable; this tightens black-box audits.
Key results
- At theoretical epsilon=10, worst-case initialization yields epsilon_emp=6.48 (MNIST) and 4.96 (CIFAR-10) on full datasets vs 3.41 and 0.69 with average-case initialization.
- On 1,000-record samples at epsilon=10, epsilon_emp reaches 7.21 (MNIST) and 6.95 (CIFAR-10).
- Last-layer-only fine-tuning of WRN-28-10 gives epsilon_emp=9.33 at epsilon=10 with worst-case init vs 9.19 with average-case init.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use worst-case rather than average-case initial parameters | DP-SGD full-model audit on MNIST · MNIST, ε=10.0 | ε_emp: 3.41 -> 6.48 | Helped |
| Use worst-case rather than average-case initial parameters | DP-SGD full-model audit on CIFAR-10 · CIFAR-10, ε=10.0 | ε_emp: 0.69 -> 4.96 | Helped |
| Use worst-case rather than average-case initial parameters | DP-SGD full-model audit · MNIST and CIFAR-10, ε=1.0 and ε=2.0 | ε_emp: 0.00, 0.06 -> 0.08, 0.54; 0.07, 0.02 -> 0.00, 0.18 | Mixed |
| Use worst-case rather than average-case initial parameters | DP-SGD last-layer fine-tuning audit · CIFAR-10, ε=10.0 | ε_emp: 9.19 -> 9.33 | Helped |
| Increase pre-training from 0 to 5 epochs to craft initial parameters (ablation) | DP-SGD audit · MNIST, ε=10.0 | ε_emp: 3.41 -> 6.48 | Helped |
| Increase pre-training from 0 to 5 epochs to craft initial parameters (ablation) | DP-SGD audit · MNIST, ε=10.0 | average clipped gradient norm: 1.00 to 0.51 | Helped |
| Reduce dataset size from the full dataset to n=100 (ablation) | DP-SGD audit · MNIST, ε=10.0 | ε_emp: 6.48 -> 7.41 | No clear effect |
| Reduce dataset size from the full dataset to n=100 (ablation) | DP-SGD audit · CIFAR-10, ε=10.0 | ε_emp: 4.96 -> 7.01 | Helped |
| Increase dataset size (ablation) | DP-SGD full-model audit · MNIST and CIFAR-10 | ε_emp | Hurt |
| Increase gradient clipping norm (ablation) | DP-SGD full-model audit · MNIST and CIFAR-10 | ε_emp | Hurt |
| Increase clipping norm from C=1.0 to C=10.0 (ablation) | DP-SGD on MNIST · MNIST, ε=1.0, 2.0, 4.0, 10.0 | accuracy: 97.2%, 97.4%, 97.5%, 97.5% -> 85.6%, 93.1%, 95.0%, 96.1% | Hurt |
| Vary dataset size and clipping norm (ablation) | DP-SGD last-layer fine-tuning audit · CIFAR-10 | ε_emp | No clear effect |
Surprising findings
- Maximum epsilon_emp over five runs slightly exceeds theoretical epsilon in last-layer fine-tuning at small epsilon, but this lies within one standard deviation.
Limitations
- High computational cost: hundreds of models must be trained for black-box auditing.
- Only full-batch gradient descent (no subsampling) is audited; sub-sampled DP-SGD remains open.
- Lower utility on CIFAR-10 (~50%) due to shallow CNNs and few iterations used.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet-32
Benchmarks: MNIST CIFAR-10
Metrics: empirical epsilon (epsilon_emp) test accuracy FPR/FNR via Clopper-Pearson bounds
Baselines: average-case initialization (Xavier) De et al. black-box audit Nasr et al. audit Jagielski et al. audit
Builds on: DP-SGD Gaussian differential privacy Clopper-Pearson confidence intervals Nasr et al. tight auditing Jagielski et al. auditing
Closest papers by meaning
- Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (ICLR 2025 · similarity 0.92)
- Auditing Private Prediction (ICML 2024 · similarity 0.91)
- The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD (ICLR 2025 · similarity 0.89)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.88)
- Auditing $f$-differential privacy in one run (ICML 2025 · similarity 0.88)
- One-shot Empirical Privacy Estimation for Federated Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.