Research map / Federated learning and privacy / Privacy attacks and secure learning

DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer

ICLR 2024 · Federated learning and privacy › Privacy attacks and secure learning · approach: Encryption-based private learning · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DP-OPT, which tunes a discrete prompt privately on the client with a local Vicuna-7b via a differentially private ensemble of in-context prompt generation, then transfers it to cloud LLMs such as DaVinci-003.

Problem

Prompt tuning on private data requires sending sensitive data to cloud LLM providers, and the resulting prompts can leak private information; hosting or exposing cloud model parameters is also undesirable.

Approach

Generate discrete instructional prompts locally with a small LLM (DLN-1 style forward/backward prompt generation), privatize the generation by sample-and-aggregate voting over disjoint demonstration subsets with a LimitedDomain mechanism, and privatize prompt selection with the exponential mechanism; the resulting prompt is transferred to cloud models without parameter sharing.

Key results

• DP-OPT with epsilon=8 on Vicuna-7b achieves 81.4% average test accuracy across four tasks versus 82.9% for non-private ICL on DaVinci-003 and 70.5% for private PromptDPSGD.
• Prompts tuned on Vicuna-7b via DLN-1 transfer to DaVinci-003 with an 8% average accuracy gain, e.g., Disaster 61.7 to 77.0.
• Under LiRA membership inference, DP-OPT AUCs stay near 0.5 (0.456 to 0.518) while DLN-1 reaches 0.772 on Mpqa and 0.773 on SST-2.

Measured effects reported in the paper

Surprising findings

• DLN-1 prompts transfer positively to larger models, outperforming their performance on the source model (positive rather than negative transfer), in contrast to PEZ soft-prompt transfer which loses accuracy.
• Instructing the LLM to avoid copying examples (privatization instruction) still leaked private samples, with LiRA AUC around 73%.

Limitations

• Experiments limited to four sentiment and classification tasks with sentence-level data.
• Privacy-utility trade-off: small epsilon limits the number of generated tokens, reducing accuracy on Vicuna-7b (about 86% at epsilon=1 on SST-2).
• Trec result for DLN-1 on Vicuna-7b is lower than reported in prior work, attributed to the local model's limitations.

Experimental setup

Datasets: SST-2 Trec Mpqa Disaster

Benchmarks: SST-2 Trec Mpqa Disaster

Metrics: test accuracy membership inference AUC (LiRA)

Baselines: ICL DLN-1 PromptSGD PromptDPSGD 0-shot Private DLN-1 PEZ LiRA membership inference

Builds on: Deep Language Network (DLN) Automatic Prompt Engineering (APE) PEZ DPSGD LimitedDomain mechanism

Closest papers by meaning

• Efficient and Privacy-Preserving Soft Prompt Transfer for LLMs (ICML 2025 · similarity 0.92)
• Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.90)
• Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.90)
• DPZero: Private Fine-Tuning of Language Models without Backpropagation (ICML 2024 · similarity 0.90)
• Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning (ICLR 2025 · similarity 0.89)
• Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.88)