Research map / Federated learning and privacy / Differential privacy

The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD

ICLR 2025 · Federated learning and privacy › Differential privacy · approach: DP-SGD refinements · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a linearized heuristic privacy analysis of DP-SGD when only the last iterate is released, validates it against privacy auditing on CIFAR-10 and Gemma 2 finetuning, and builds counterexamples where it underestimates leakage.

Problem

Standard composition-based DP-SGD analysis assumes adversaries see all intermediate iterates, which is loose when only the final model is released; existing auditing gives lower bounds with a large gap to upper bounds in this setting.

Approach

Under a linear loss and regularizer, the last iterate reduces to a one-dimensional distinguishing problem between Gaussian mixtures (Binomial(T,q) plus Gaussian noise vs. Gaussian noise), giving an exact closed-form privacy accountant that serves as a heuristic for nonlinear deep models; it is always at most the all-iterates bound.

Key results

• Heuristic epsilon for Gemma 2 DP finetuning on PersonaChat is about 6.0 across batch sizes 512-2048, while empirical epsilon from auditing is about 2.0-2.1 and standard epsilon ranges from 8 to 26.
• On CIFAR-10 with a WideResNet, batch size 4096 gives heuristic epsilon 6.34 and 79.5% accuracy, versus standard epsilon 8, showing comparable accuracy at smaller batch sizes (512: 79.9%, heuristic 7.0).
• Gradient space attacks with other examples sampled from data yield empirical epsilon well below the heuristic, while a malicious-dataset attack that encodes all iterates in the final parameters matches the standard analysis and exceeds the heuristic.

Measured effects reported in the paper

Surprising findings

• Heuristic epsilon is not monotone in the number of steps T, since more steps add both noise and gradient signal.
• Input space attacks achieve tighter empirical auditing than gradient space attacks in the natural-data setting, contrary to prior work.

Limitations

• Heuristic relies on linearity assumption and can be exceeded by contrived counterexamples (zeroed-out regularizer, pathological regularizer, malicious dataset).
• Heuristic is not a rigorous privacy guarantee and cannot replace formal analysis.
• Language model auditing uses vocabulary tokens absent from data as canaries, and optimal canary design for LMs is left open.

Experimental setup

Datasets: CIFAR-10 PersonaChat

Benchmarks: CIFAR-10 PersonaChat

Metrics: epsilon empirical epsilon from privacy auditing test accuracy perplexity

Baselines: standard composition-based DP-SGD analysis full-batch DP-GD analysis gradient space attacks input space attacks

Builds on: DP-SGD privacy auditing with canaries PLD accounting Gaussian differential privacy

Closest papers by meaning

• Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (ICLR 2025 · similarity 0.91)
• Nearly Tight Black-Box Auditing of Differentially Private Machine Learning (NeurIPS 2024 · similarity 0.89)
• Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness (ICLR 2025 · similarity 0.89)
• Shifted Interpolation for Differential Privacy (ICML 2024 · similarity 0.89)
• Auditing Private Prediction (ICML 2024 · similarity 0.89)
• Scalable DP-SGD: Shuffling vs. Poisson Subsampling (NeurIPS 2024 · similarity 0.88)