AI Paper Atlas

Research map / Federated learning and privacy / Differential privacy

Differentially Private Preference Data Synthesis for Large Language Model Alignment

Fengyu Gao, Jing Yang

ICML 2026 · Federated learning and privacy › Differential privacy · approach: Differentially private synthetic data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune on DP synthetic preference data instead of real private data with DP-FTPythia-2.8B DPO · Anthropic-HH epsilon=2GPT-4o win rate (%): 37.02 -> 56.48Improved
use DPPrefSyn synthetic data instead of non-private real dataPythia-2.8B DPO · OpenAssistant epsilon=2 vs DP-FT epsilon=infinityGPT-4o win rate (%): 8.20 -> 11.04Improved
add DPO on top of SFT with synthetic data (ablation)DPPrefSyn · Anthropic-HH epsilon=2GPT-4o win rate (%): 55.21 -> 56.48Improved
tighten privacy budget from epsilon=8 to epsilon=0.5 (ablation)DPPrefSyn SFT+DPO · OpenAssistantGPT-4o win rate (%): 12.05 -> 9.86Hurt
tighten DP from epsilon infinity to 4 on Llama-3-8B DPO (ablation)DP-FT vs DPPrefSyn · OpenAssistant, Llama-3-8BGPT-4o win rate (%): DP-FT 59.13 -> 45.79; DPPrefSyn 61.52 -> 60.25Mixed
use PPO with averaged K cluster reward models instead of DPO (ablation)DPPrefSyn epsilon=4 · OpenAssistantGPT-4o win rate (%): 11.85 (DPO) -> 10.11 (PPO)Hurt
use a stronger generator LLM (Qwen3-4B-Instruct instead of Llama-3-8B-Instruct) (ablation)DPPrefSyn · OpenAssistant epsilon=4, Qwen-3-4BGPT-4o win rate (%): 51.69 -> 86.52Improved
increase public prompts from 5K to 52K (ablation)DPPrefSyn · OpenAssistant epsilon=4GPT-4o win rate after SFT+DPO (%): 9.44 -> 11.85Improved
remove DP-KMeans (single cluster) or DP-PCA (ablation)DPPrefSyn · OpenAssistant epsilon=4GPT-4o win rate (%)Improved
increase number of clusters K from 1 to 3, then 5 and 8 (ablation)DPPrefSyn · OpenAssistant epsilon=4GPT-4o win rate (%)Improved
set quality filter threshold to 0.5, versus none or 1 (ablation)DPPrefSyn · OpenAssistantGPT-4o win rate (%)Mixed
use DPPrefSyn data instead of DP-FT for downstream modelmembership inference attack on DPO-tuned models · OpenAssistant, Qwen-3-4B, epsilon=4MIA AUC (%): 70.15 -> 50.99Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy