Research map / Federated learning and privacy / Differential privacy
Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a differentially private method that generates synthetic few-shot ICL demonstrations from private data by privately aggregating per-token LLM next-token distributions over disjoint subsets, evaluated with GPT-3 models on text classification and slot-filling tasks.
Problem
In-context learning with LLMs on private datasets risks leaking demonstrated private examples via prompt extraction or regurgitation, and heuristic PII removal does not give formal privacy guarantees.
Approach
Generate synthetic demonstrations token by token: at each step, sample disjoint subsets of private data, get next-token distributions from the LLM conditioned on each subset, and aggregate them with Gaussian or Report-Noisy-Max/exponential noise (PATE-style), with subsampling amplification and numerical composition. Since demonstrations are generated once, any number of queries afterward incur no
Key results
- On TREC, 4-shot DP generations at epsilon=1 reach 50.7% accuracy with GPT-3 Babbage, versus 35.4% for fully private 0-shot and 50.6% for the non-private baseline.
- On AGNews, instruction-only synthetic 4-shot demonstrations (epsilon=0, no private data) reach 68.0% versus 47.9% zero-shot, close to the non-private 69.3%.
- Membership inference AUC drops from 81.84 for non-private 1-shot ICL to about 50.5 for DP generations at all tested epsilon values.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate DP synthetic demonstrations from private data rather than use no demonstrations | GPT-3 Babbage ICL · AGNews test | accuracy: 47.9_{0.0} -> 64.1_{3.9} | Helped |
| Generate DP synthetic demonstrations from private data rather than use no demonstrations | GPT-3 Babbage ICL · DBPedia test | accuracy: 30.4_{0.0} -> 81.2_{3.0} | Helped |
| Generate DP synthetic demonstrations from private data rather than use no demonstrations | GPT-3 Babbage ICL · TREC test | accuracy: 35.4_{0.0} -> 50.7_{4.1} | Helped |
| Generate DP synthetic demonstrations from private data rather than use no demonstrations | GPT-3 Babbage ICL · MIT-G test | accuracy: 17.2_{0.0} -> 46.3_{7.8} | Helped |
| Generate DP synthetic demonstrations from private data rather than use no demonstrations | GPT-3 Babbage ICL · MIT-D test | accuracy: 47.9_{0.0} -> 69.2_{10.0} | Helped |
| Generate demonstrations from private data instead of instruction-only demonstrations | GPT-3 Babbage ICL · TREC test | accuracy: 45.7_{4.0} -> 50.7_{4.1} | Helped |
| Generate demonstrations from private data instead of instruction-only demonstrations | GPT-3 Babbage ICL · DBPedia test | accuracy: 60.4_{10.7} -> 81.2_{3.0} | Helped |
| Use DP few-shot generation rather than the non-private 4-shot solution | GPT-3 Babbage ICL · MIT-D test | accuracy: 80.1_{0.7} vs 74.6_{4.2} | Hurt |
| Vary demonstration count from 4-shot to 1-, 2-, 8-, or 12-shot (ablation) | DP few-shot, ε=4 · DBPedia test | accuracy: 83.1_{4.3}; 75.4_{4.2}; 76.0_{5.7}; 82.7_{1.0}; 80.7_{2.4} | Mixed |
| Use 4-shot instruction-generated demonstrations instead of 0-shot | GPT-3 Babbage ICL · AGNews test | accuracy: 47.9_{0.0} -> 68.0_{0.8} | Helped |
| Replace Gaussian with Exponential mechanism (ablation) | DP few-shot, ε=4 · DBPedia test | accuracy: 83.1_{4.3} -> 80.9_{4.8} | Hurt |
| Increase model size from Babbage to Davinci (ablation) | DP few-shot, ε=4 · DBPedia test | accuracy: 83.1_{4.3} -> 87.2_{2.6} | Helped |
Surprising findings
- Instruction-only generated demonstrations (no private data) substantially improve zero-shot accuracy and nearly match non-private ICL on AGNews.
- For 8- and 12-shot settings on DBPedia, random private demonstrations outperform the DP generated ones.
Limitations
- Each generated token resamples private demonstrations, so privacy budget composes over many iterations, requiring high noise or a limited number of steps.
- Relies on a trusted LM that directly accesses private data during generation; only GPT-3 models via OpenAI API were evaluated.
- Gains diminish for larger shot counts (8 and 12) where random private samples perform better.
Experimental setup
Datasets: AGNews DBPedia TREC MIT Movies trivia10k13
Benchmarks: AGNews DBPedia TREC MIT Movies trivia10k13 (genre and director slots)
Metrics: accuracy
Baselines: zero-shot ICL non-private 4-shot ICL with random private samples non-private ICL with Alg. generations (sigma=0)
Builds on: PATE DP-SGD Permute-and-Flip / Report Noisy Max numerical composition of privacy curves
Closest papers by meaning
- Privacy-Preserving In-Context Learning for Large Language Models (ICLR 2024 · similarity 0.93)
- Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning (ICLR 2025 · similarity 0.92)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.88)
- Plausible Token Amplification for Improving Accuracy of Differentially Private In-Context Learning Based on Implicit Bayesian Inference (ICML 2025 · similarity 0.87)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.87)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.