Research map / Federated learning and privacy / Differential privacy
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Fengyu Gao, Jing Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on DP synthetic preference data instead of real private data with DP-FT | Pythia-2.8B DPO · Anthropic-HH epsilon=2 | GPT-4o win rate (%): 37.02 -> 56.48 | Improved |
| use DPPrefSyn synthetic data instead of non-private real data | Pythia-2.8B DPO · OpenAssistant epsilon=2 vs DP-FT epsilon=infinity | GPT-4o win rate (%): 8.20 -> 11.04 | Improved |
| add DPO on top of SFT with synthetic data (ablation) | DPPrefSyn · Anthropic-HH epsilon=2 | GPT-4o win rate (%): 55.21 -> 56.48 | Improved |
| tighten privacy budget from epsilon=8 to epsilon=0.5 (ablation) | DPPrefSyn SFT+DPO · OpenAssistant | GPT-4o win rate (%): 12.05 -> 9.86 | Hurt |
| tighten DP from epsilon infinity to 4 on Llama-3-8B DPO (ablation) | DP-FT vs DPPrefSyn · OpenAssistant, Llama-3-8B | GPT-4o win rate (%): DP-FT 59.13 -> 45.79; DPPrefSyn 61.52 -> 60.25 | Mixed |
| use PPO with averaged K cluster reward models instead of DPO (ablation) | DPPrefSyn epsilon=4 · OpenAssistant | GPT-4o win rate (%): 11.85 (DPO) -> 10.11 (PPO) | Hurt |
| use a stronger generator LLM (Qwen3-4B-Instruct instead of Llama-3-8B-Instruct) (ablation) | DPPrefSyn · OpenAssistant epsilon=4, Qwen-3-4B | GPT-4o win rate (%): 51.69 -> 86.52 | Improved |
| increase public prompts from 5K to 52K (ablation) | DPPrefSyn · OpenAssistant epsilon=4 | GPT-4o win rate after SFT+DPO (%): 9.44 -> 11.85 | Improved |
| remove DP-KMeans (single cluster) or DP-PCA (ablation) | DPPrefSyn · OpenAssistant epsilon=4 | GPT-4o win rate (%) | Improved |
| increase number of clusters K from 1 to 3, then 5 and 8 (ablation) | DPPrefSyn · OpenAssistant epsilon=4 | GPT-4o win rate (%) | Improved |
| set quality filter threshold to 0.5, versus none or 1 (ablation) | DPPrefSyn · OpenAssistant | GPT-4o win rate (%) | Mixed |
| use DPPrefSyn data instead of DP-FT for downstream model | membership inference attack on DPO-tuned models · OpenAssistant, Qwen-3-4B, epsilon=4 | MIA AUC (%): 70.15 -> 50.99 | Improved |
Closest papers by meaning
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.90)
- Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment (ICML 2025 · similarity 0.90)
- Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models (ICML 2025 · similarity 0.89)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.89)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.89)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
