Research map / Federated learning and privacy / Differential privacy
Empirical Privacy Variance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that DP-SGD/DP-Adam fine-tunes of GPT-2 and Llama-2 calibrated to the same (ε,δ) guarantee differ widely in memorization-based empirical privacy, and regression plus heuristics show better utility-tuned configs tend to leak more.
Problem
Models calibrated to the same (ε,δ)-DP guarantee via different DP-SGD hyperparameters may have very different empirical privacy (memorization of secrets), so ε alone does not characterize privacy risk in LLM fine-tuning.
Approach
Defines empirical privacy variance: under identical DP guarantees, hyperparameter configurations yield widely varying memorization scores; regression on batch size, iterations and learning rate reveals a no-free-lunch trade-off, and hyperparameter selection heuristics that account for empirical privacy reduce relative privacy risk.
Key results
- Models with the same (ε,δ)-DP guarantee show substantial empirical privacy variance; e.g., Llama-2-7b on TOFU-4 at ε=8 has AIR either above 0.8 or near-zero knowledge of secrets.
- Hyperparameter choices that improve utility (larger batch, higher learning rate, more iterations) worsen empirical privacy, a no-free-lunch trade-off with regression coefficients all positive.
- The proposed sequential selection procedure consistently lowers relative privacy risk versus best-utility selection across settings and ε values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase batch size while holding other hyperparameters fixed | DP-SGD fine-tuning · Enron (GPT-2-S) and TOFU (Llama-2-7b) | ACR / AIR: 0.13$^{***}$; 0.029$^{***}$ | Hurt |
| Increase training iterations while holding other hyperparameters fixed | DP-SGD fine-tuning · Enron (GPT-2-S) and TOFU (Llama-2-7b) | ACR / AIR: 0.37$^{***}$; 0.048$^{***}$ | Hurt |
| Increase learning rate while holding other hyperparameters fixed | DP-SGD fine-tuning · Enron (GPT-2-S) and TOFU (Llama-2-7b) | ACR / AIR: 0.51$^{***}$; 0.068$^{***}$ | Hurt |
| Increase compute while modeling learning rate separately | DP-SGD fine-tuning · Enron (GPT-2-S) and TOFU (Llama-2-7b) | ACR / AIR: 0.22$^{***}$; 0.039$^{***}$ | Hurt |
| Increase learning rate while modeling compute | DP-SGD fine-tuning · Enron (GPT-2-S) and TOFU (Llama-2-7b) | ACR / AIR: 0.53$^{***}$; 0.066$^{***}$ | Hurt |
| At fixed compute and learning rate, use a larger batch size and proportionally fewer iterations | DP-SGD fine-tuning · Enron (GPT-2-S), ACR | ACR | Helped |
| At fixed updates, use a smaller learning rate and proportionally greater compute | DP-SGD fine-tuning · Enron (GPT-2-S), ACR | ACR | Helped |
| Use full fine-tuning rather than LoRA | DP fine-tuning · GPT-2/Enron and Llama-2/TOFU experiments | Empirical privacy variance | Hurt |
| Increase model size | DP fine-tuning · GPT-2 models on Enron | Empirical privacy variance | Hurt |
| Increase dataset size or secret density | DP fine-tuning · TOFU variants | Empirical privacy variance | Hurt |
| Increase ε | DP fine-tuning · Enron and TOFU experiments | Empirical privacy variance | Hurt |
| Evaluate the maximum rather than the mean across secrets (ablation) | Empirical privacy analysis · Experiments across datasets and models | Qualitative persistence of key results | No clear effect |
Surprising findings
- Estimated audited epsilon (black-box auditing) is nearly uncorrelated with empirical privacy score (Spearman -0.13), so loss-based auditing may fail to capture memorization-based empirical privacy.
- Under the PRV accountant, larger T can yield smaller ε in the small-δ regime, conflicting with the expectation that more iterations weaken privacy.
Limitations
- Empirical privacy measured only through memorization of a small curated set of secrets (13 Enron, 52 TOFU), averaged rather than worst-case.
- Regression is exploratory linear analysis and the heuristics are pairwise and not guaranteed to hold universally.
- Hyperparameter tuning privacy cost is not accounted for, and shuffled batches are used instead of Poisson subsampling.
Experimental setup
Datasets: Enron Email TOFU
Benchmarks: Enron Email TOFU
Metrics: adversarial compression ratio verbatim memorization ratio attribute inference ratio negative log likelihood Spearman correlation relative privacy risk
Baselines: best-utility selection standalone worst-utility heuristic
Builds on: DP-SGD DP-Adam adversarial compression ratio Greedy Coordinate Gradient PRV accountant
Closest papers by meaning
- Auditing $f$-differential privacy in one run (ICML 2025 · similarity 0.89)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.88)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.88)
- Auditing Private Prediction (ICML 2024 · similarity 0.88)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.87)
- Beyond the Calibration Point: Mechanism Comparison in Differential Privacy (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.