Research map / Federated learning and privacy / Differential privacy

Plausible Token Amplification for Improving Accuracy of Differentially Private In-Context Learning Based on Implicit Bayesian Inference

ICML 2025 · Federated learning and privacy › Differential privacy · approach: Differentially private synthetic data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Plausible Token Amplification (PTA), which reweights next-token distributions by the private-to-public likelihood ratio before Gaussian noising, giving a Bayesian-inference theory for DP-ICL and improved accuracy over Tang et al. (2024).

Problem

DP-ICL built from DP synthetic demonstrations loses accuracy as noise grows; the vocabulary-limiting heuristic of Tang et al. (2024) lacks theoretical grounding and does not fully exploit amplifying task-distinctive tokens.

Approach

Interpret ICL as implicit Bayesian inference to derive a DP-ICL distinguishability condition, then modify next-token probabilities by amplifying tokens whose private-vs-public likelihood ratio is high (with a KL-regularized base distribution), using public prompts so no extra privacy cost is incurred.

Key results

• On AGNews at epsilon=1, PTA (P1) reaches 87.88±1.11% vs 83.74±1.90% for the Tang et al. baseline (B1) with Mistral-7B.
• On GINC with GPT-2 16-layer at epsilon=1, PTA achieves 93.99±1.35% vs 90.80±2.79% for B1.
• PTA improves the proportion of generated demonstrations satisfying the distinguishability condition on GINC (49.33% vs 44.89% at epsilon=1).

Measured effects reported in the paper

Surprising findings

• PTA without the KL/base-probability regularization (PA3) performs best on DBPedia at epsilon=1, suggesting the KL term can limit useful adaptation in some settings.
• Top-p truncation variant (PA2) performs notably worse than PTA on GINC, since low-probability tokens matter with small vocabularies.

Limitations

• Theoretical analysis relies on Assumptions 1-4 from Xie et al. (2022) and approximations (e.g., dropping the second term of the likelihood ratio decomposition).
• Accuracy gains are not uniform: PTA underperforms on DBPedia at epsilon=1 and with weaker base models such as GPT-2 4-layer and Llama2-7B-GPTQ.

Experimental setup

Datasets: GINC AGNews DBPedia TREC

Benchmarks: GINC AGNews DBPedia TREC

Metrics: ICL accuracy Pr[RHS of Lemma 1 negative] (distinguishability proportion)

Baselines: Tang et al. 2024 (B1) Baseline with top-p (BA2) 4-shot ICL with original private demonstrations (R0)

Builds on: Tang et al. 2024 DP-ICL synthetic demonstrations Xie et al. 2022 implicit Bayesian ICL Gopi et al. 2021 numerical PRV accountant Zhao et al. 2021 calibrate before use

Closest papers by meaning

• Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning (ICLR 2025 · similarity 0.88)
• Privacy-Preserving In-Context Learning for Large Language Models (ICLR 2024 · similarity 0.88)
• Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation (ICLR 2024 · similarity 0.87)
• SynBench: A Benchmark for Differentially Private Text Generation (NeurIPS 2026 · similarity 0.84)
• Towards Auto-Regressive Next-Token Prediction: In-context Learning Emerges from Generalization (ICLR 2025 · similarity 0.84)
• Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.84)