Research map / Federated learning and privacy / Differential privacy
Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaDPSyn, a data-adaptive differentially private prompt synthesis method that uses Precision-Focused Iterative Radius Reduction to reduce aggregation noise when generating synthetic few-shot demonstrations for in-context learning.
Problem
Protecting private examples used as demonstrations in in-context learning while keeping high ICL accuracy; prior DP few-shot generation uses data-independent noise that is larger than necessary.
Approach
Adaptively shrinks the aggregation radius under DP (starting from the simplex radius, using a GoodRadius target and iterative coverage checks), projecting next-token probability vectors onto a ball so that the Gaussian noise scales with the small radius rather than the simplex radius.
Key results
- At epsilon=1 on AGNews, AdaDPSyn reaches 65.42% accuracy vs 60.74% for DP few-shot generation (non-private 65.92%).
- At epsilon=1 on MIT-G, AdaDPSyn reaches 37.59% vs 31.15% for DP few-shot generation.
- At epsilon=1 on TREC with GPT-4o mini, AdaDPSyn reaches 69.32% vs 62.48% for DP few-shot generation (non-private 72.32%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AdaDPSyn instead of DP few-shot generation | 4-shot ICL on AGNews at ε=1 · AGNews test | accuracy: 60.74 -> 65.42 | Helped |
| use AdaDPSyn instead of DP few-shot generation | 4-shot ICL on DBPedia at ε=1 · DBPedia test | accuracy: 64.68 -> 66.76 | Helped |
| use AdaDPSyn instead of DP few-shot generation | 4-shot ICL on TREC at ε=1 · TREC test | accuracy: 62.48 -> 69.32 | Helped |
| use AdaDPSyn instead of DP few-shot generation | 4-shot ICL on MIT-G at ε=1 · MIT-G test | accuracy: 31.15 -> 37.59 | Helped |
| use AdaDPSyn instead of DP few-shot generation | 4-shot ICL on MIT-D at ε=1 · MIT-D test | accuracy: 72.87 -> 73.35 | Helped |
| increase demonstrations from 1-shot to 8-shot (ablation) | AdaDPSyn on DBPedia at ε=4 · DBPedia test | accuracy: ${65.94}_{0.27}$ -> ${67.55}_{1.27} | Helped |
| replace Llama-2-7B-hf with GPT-3.5 Turbo for downstream ICL (ablation) | AdaDPSyn on DBPedia at ε=4 · DBPedia test | accuracy: $67.12_{1.09}$ -> $91.92_{1.46}$ | Helped |
| replace Llama-2-7B-hf with GPT-4o mini for downstream ICL (ablation) | AdaDPSyn on DBPedia at ε=4 · DBPedia test | accuracy: $67.12_{1.09}$ -> $93.08_{1.09}$ | Helped |
| vary λ and T̂ across the tested values (ablation) | AdaDPSyn on AGNews at ε=8 · AGNews test | accuracy: $64.28_{1.48}$–$65.92_{1.02}$ | No clear effect |
| vary λ and T̂ across the tested values (ablation) | AdaDPSyn on DBPedia at ε=8 · DBPedia test | accuracy: $65.28_{3.13}$–$67.70_{1.68}$ | No clear effect |
| vary λ and T̂ across the tested values (ablation) | AdaDPSyn on TREC at ε=8 · TREC test | accuracy: $70.08_{2.53}$–$72.08_{2.24}$ | Mixed |
| vary λ and T̂ across the tested values (ablation) | AdaDPSyn on MIT-G at ε=8 · MIT-G test | accuracy: $36.33_{7.28}$–$38.31_{5.03}$ | Mixed |
Surprising findings
- The fully private epsilon=0 instruction-only baseline outperforms DP few-shot generation at epsilon 1 and 2 on AGNews.
Limitations
- No theoretical utility guarantee for ICL accuracy
- Results on TREC with Llama-2-7b show high variance, requiring GPT-4o mini for downstream ICL
- Relies on a fixed set of hyperparameters borrowed from the baseline without separate tuning
Experimental setup
Datasets: AGNews DBPedia TREC MIT Movies trivia10k13 (MIT-G, MIT-D)
Benchmarks: AGNews DBPedia TREC MIT-G MIT-D
Metrics: accuracy
Baselines: DP few-shot generation random 4-shot demonstrations from private data instruction-only synthetic demonstrations (epsilon=0)
Builds on: DP few-shot generation PATE-style sample-and-aggregate GoodRadius Rényi differential privacy
Closest papers by meaning
- Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation (ICLR 2024 · similarity 0.92)
- Privacy-Preserving In-Context Learning for Large Language Models (ICLR 2024 · similarity 0.90)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.89)
- Plausible Token Amplification for Improving Accuracy of Differentially Private In-Context Learning Based on Implicit Bayesian Inference (ICML 2025 · similarity 0.88)
- Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs (ICML 2025 · similarity 0.88)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.