Research map / Federated learning and privacy / Differential privacy

Privacy-Preserving In-Context Learning for Large Language Models

ICLR 2024 · Federated learning and privacy › Differential privacy · approach: Differentially private synthetic data · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a differentially private in-context learning method (DP-ICL) that privately selects top-k tokens from exemplar outputs via a Gumbel-noise exponential mechanism and propose-test-release, with RDP and subsampling analysis.

Problem

LLM in-context learning can leak sensitive private information contained in in-context exemplars; the paper targets privacy-preserving ICL with formal differential privacy guarantees.

Approach

Privately releases the top-k most frequent tokens across exemplar-conditioned outputs by combining a report-noisy-max style selection (Gumbel noise on count gaps) with propose-test-release, and gives RDP and approximate-RDP privacy accounting including amplification by subsampling.

Key results

• Shows the private top-k selection via Gumbel noise (sensitivity 2) is epsilon-DP, with an RDP bound eps_EM(alpha) = min(alpha*eps^2/2, ...).
• Proves the propose-test-release top-k algorithm is delta-approximate alpha/(2 sigma^2)-RDP.
• Provides a subsampling amplification theorem for approximate RDP with effective rate q(1-delta)/(1-q*delta).

Limitations

• The provided text contains only the algorithm and privacy analysis sections; no experimental evaluation is included in the extract.
• Several theorem statements are incomplete or contain placeholder references ([ref], [cite]), so the analysis cannot be fully verified from the extract.

Experimental setup

Builds on: exponential mechanism report-noisy-max propose-test-release Renyi differential privacy

Closest papers by meaning

• Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation (ICLR 2024 · similarity 0.93)
• Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning (ICLR 2025 · similarity 0.90)
• Plausible Token Amplification for Improving Accuracy of Differentially Private In-Context Learning Based on Implicit Bayesian Inference (ICML 2025 · similarity 0.88)
• Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.88)
• Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.87)
• DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.87)