Research map / Large language models / Instruction and fine-tuning
Privacy-Preserving Instructions for Aligning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates differentially private synthetic user instructions by DP-fine-tuning LLaMA generators and resampling them with a DP histogram filter so they match real Chatbot Arena instruction distributions, then uses them for SFT and PPO-based RLHF.
Problem
User instructions collected by LLM services contain sensitive information that is exposed to human annotators during data labeling and can be memorized by aligned models; existing DP fine-tuning only addresses memorization.
Approach
Replace real user instructions with DP synthetic instructions from a privately fine-tuned generator, then apply a novel DP histogram-based resampling step (k-means clusters of synthetic embeddings, noisy histogram of real data) to align the synthetic distribution with real instructions at small extra privacy cost.
Key results
- DP synthetic instructions with filtering (eps=5.98) give 67.8% win-rate for 7B SFT vs 60.7% for real Chatbot Arena instructions under DP (eps=5.94) and 68.9% for non-private real data.
- Filtering improves the 7B SFT win-rate by 8.6% relative over unfiltered DP synthetic data (eps=5.94 unfiltered 62.7%, filtered eps=2.91 65.6%).
- MAUVE (unigram) improves from 0.932 to 0.958 at eps=2.86 to 2.91 after filtering, and from 0.942 to 0.961 at eps=5.94 to 5.98.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Resample initial DP synthetic instructions with the private histogram | DP synthetic instructions generated from LLaMA 13B · Chatbot Arena instructions | MAUVE unigram: 0.942 -> 0.961 | Helped |
| Resample initial DP synthetic instructions with the private histogram | DP synthetic instructions generated from LLaMA 13B · Chatbot Arena instructions | MAUVE Sentence-T5: 0.912 -> 0.975 | Helped |
| Resample initial DP synthetic instructions with the private histogram | DP synthetic reviews generated from GPT2-Large · IMDB reviews | MAUVE Sentence-T5: 0.647 -> 0.825 | Helped |
| Resample initial DP synthetic abstracts with the private histogram | DP synthetic abstracts generated from GPT2-Large · PubMed abstracts | MAUVE Sentence-T5: 0.594 -> 0.811 | Helped |
| Use filtered rather than unfiltered DP synthetic instructions | LLaMA 7B supervised fine-tuning · Chatbot Arena instructions (test) | win-rate: 62.7% (± 0.34) -> 65.6% (±0.36) | Helped |
| Use filtered DP synthetic instructions instead of DP training on real instructions | LLaMA 7B supervised fine-tuning · Chatbot Arena instructions (test) | win-rate: 60.7% (± 0.42) -> 67.8% (± 0.32) | Helped |
| Use in-domain Chatbot Arena instructions instead of out-of-domain FLAN instructions | LLaMA 7B supervised fine-tuning · Chatbot Arena instructions (test) | win-rate: 50% -> 68.9% (± 0.31) | Helped |
| Increase the gradient clipping threshold from C=0.01 to C=0.5 (ablation) | DP instruction generator · Chatbot Arena instructions | MAUVE unigram: 0.667 -> 0.839 | Helped |
| Increase the gradient clipping threshold from C=0.01 to C=0.5 (ablation) | DP instruction generator · Chatbot Arena instructions | MAUVE Sentence-T5: 0.843 -> 0.890 | Helped |
| Use LLaMA 13B rather than LLaMA 7B as the pretrained generator (ablation) | DP instruction generator · Chatbot Arena instructions | MAUVE unigram: 0.839 -> 0.942 | Helped |
| Use LLaMA 13B rather than LLaMA 7B as the pretrained generator (ablation) | DP instruction generator · Chatbot Arena instructions | MAUVE Sentence-T5: 0.890 -> 0.912 | Helped |
| Use 500 thousand rather than one million initial synthetic instructions (ablation) | Private histogram resampling · Chatbot Arena instructions, large K | MAUVE | Hurt |
Surprising findings
- Private real-instruction training (DP-Adam on real instructions and responses) underperforms training on DP synthetic instructions, which the authors attribute to the extra cost of privatizing long responses.
- Gradient clipping threshold strongly affects synthetic instruction length and MAUVE, with small thresholds biasing generators toward short outputs.
Limitations
- Relies on an off-the-shelf 304M reward model rather than privately trained reward model in RLHF.
- AlpacaEval instructions do not accurately represent real-world user queries; best DP models slightly underperform Vicuna-v1.3 there.
- Experiments use GPT-3.5-Turbo for answer annotation and GPT-4 judging rather than human annotators.
Experimental setup
Datasets: LMSYS-Chat-1M Chatbot Arena Conversations FLAN OpenOrca IMDB PubMed
Benchmarks: Chatbot Arena instructions test split AlpacaEval MAUVE on Chatbot Arena instructions
Metrics: win-rate MAUVE (unigram and Sentence-T5) average test reward loss rank leakage rate
Baselines: Vicuna-v1.3 LLaMA2-Chat FLAN (non-private) Chatbot Arena (non-private) Text-davinci-003
Builds on: DP-SGD DP-Adam MAUVE Private Evolution LoRA
Closest papers by meaning
- Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.91)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.90)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.89)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.89)
- Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs (ICML 2025 · similarity 0.89)
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.