Research map / Large language models / Language, safety and interpretability
Privacy Auditing of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Designs canaries (notably new-token and unigram/bigram-based canaries) that are far easier to memorize, yielding much stronger loss-based membership inference on LLMs and the first nontrivial black-box single-run DP audit of an LLM (empirical epsilon about 1 at theoretical epsilon 4).
Problem
Existing privacy audits of LLMs use random canaries that yield weak membership inference, so they give loose empirical lower bounds on privacy leakage and can underestimate memorization and DP training leakage.
Approach
Craft input-space canaries whose secret tokens are highly unlikely under the training data or model (new never-seen tokens, least-frequent unigrams, unlikely bigrams, model-based low-likelihood tokens), with random OOD prefixes, so a simple loss-threshold attack separates members from non-members much better than random canaries.
Key results
- On Qwen2.5-0.5B, new-token canaries reach 49.6% TPR at 1% FPR versus 4.2% for the prior random-canary approach.
- With DP-SGD at theoretical epsilon 4 (GPT-2 LoRA/FFT, PersonaChat/E2E), the new-token canary gives an audited epsilon of about 1.01 at 99% confidence.
- Increasing training steps from T=10 to T=100 raises the 95% audited epsilon from 0 to 0.91 in the subsampling ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random canaries with new token canaries | GPT2-small DP audit · LoRA-E2E | audited ε (95% CI): 0.13 -> 1.24 | Helped |
| replace random canaries with new token canaries | GPT2-small DP audit · FFT-E2E | audited ε (95% CI): 0.13 -> 1.04 | Helped |
| replace new token canaries with bigram canaries | GPT2-small DP audit · FFT-PersonaChat | audited ε (95% CI): 0.84 -> 1.29 | Helped |
| replace prior canaries with designed canaries | Qwen2.5-0.5B membership inference · Qwen2.5-0.5B | TPR at 1% FPR: 4.2% -> 49.6% | Helped |
| increase canary length from 1 to 8 tokens (ablation) | unigram and random canaries · canary length ablation | TPR at 1% FPR | Helped |
| replace SFT loss with NWP full-sequence loss (ablation) | membership inference · PersonaChat | TPR at 1% FPR | Hurt |
| increase learning rate from 2e-5 to 5e-5 (ablation) | Llama3.2-1B with new token canaries · PersonaChat | TPR at 1% FPR: 28.2% -> 54.6% | Helped |
| increase subsampling rate from q=0.01 to q=0.1 (ablation) | GPT2-small DP audit · DP audit | audited ε (95% CI): 0.43 -> 0.84 | Helped |
| increase training steps from T=10 to T=100 (ablation) | GPT2-small DP audit · DP audit | audited ε (95% CI): 0 -> 0.91 | Helped |
| use random token prefixes rather than test-data prefixes (ablation) | GPT2-small DP audit · DP audit | audited ε (95% CI): 0.63 -> 0.84 | Helped |
| use SFT loss over target tokens rather than NWP full-sequence loss (ablation) | GPT2-small DP audit · DP audit | audited ε (95% CI): 0.0 -> 0.84 | Helped |
| vary LoRA rank from 4 to 512 (ablation) | LoRA fine-tuning · LoRA audit | AUC: 0.753 -> 0.774 | No clear effect |
Surprising findings
- Random prefixes (OOD tokens) give better audits than in-distribution test-data prefixes for DP auditing.
- Under DP auditing, the NWP objective yields zero audited epsilon, while SFT yields 0.84 (95%).
Limitations
- Auditing relies on random OOD prefixes, which gives less semantic signal than in-distribution data.
- New-token canaries add embedding rows and slightly degrade perplexity.
- Pythia-160M could not be trained to reasonable perplexity, so its audit was poor; subsampling rate q=0.01 gives weaker audits than q=0.1.
Experimental setup
Datasets: PersonaChat E2E
Benchmarks: PersonaChat E2E
Metrics: TPR at 1% FPR empirical epsilon lower bound (95% and 99% CI) perplexity AUC
Baselines: random canary unigram canary bigram canary model-based canary
Builds on: DP-SGD one-run privacy auditing (Steinke et al.) loss-threshold membership inference random canaries
Closest papers by meaning
- The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.91)
- PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.91)
- Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.90)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.90)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.90)
- Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.