Research map / Federated learning and privacy / Differential privacy

A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO

ICML 2025 · Federated learning and privacy › Differential privacy · approach: Private online and user-level learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives a unified theoretical analysis of RLHF and DPO under local label differential privacy and adversarial label corruption (CTL vs LTC orderings), reducing offline alignment under linear/log-linear models to private, corrupted logistic regression parameter estimation.

Problem

Offline alignment (RLHF and DPO) with preference labels that are both privatized for local differential privacy and adversarially corrupted, where the order of privacy and corruption matters; prior theory was suboptimal or covered only one of privacy or corruption.

Approach

Reduces offline RLHF and DPO under linear/log-linear assumptions to logistic regression parameter estimation, then uses a debiased "shifting and scaling" log-loss on randomized-response labels that is unbiased for the standard log-loss, yielding suboptimality bounds showing LTC has an extra c(epsilon) corruption cost compared to CTL.

Key results

• Under linear models, the paper proves CTL and LTC suboptimality bounds for private and robust RLHF and DPO, with LTC carrying an extra c(epsilon) factor in the corruption term; for DPO under label corruption it gives an O(1/sqrt(n)) rate versus the O(1/n^{1/4}) rate of prior rDPO analysis.
• On GPT2-large with finance preference data, rDPO beats DPO in the private setting (e.g., epsilon=0.5: 65.8% vs 60.4% win rate).
• Under alpha=0.1 corruption, rDPO wins more often under CTL than LTC (epsilon=0.5: 64.4% vs 58.6%).

Measured effects reported in the paper

Surprising findings

• Adding random label flips at rate about 1/(e+1) in the non-private setting makes rDPO outperform DPO, resembling label smoothing.
• The authors note that prior RLHF private results with v=0 reference vectors may fail to converge as n grows, due to the choice of lambda on order 1/n.

Limitations

• Main theory relies on linear reward and log-linear policy assumptions; general neural-network results lack formal guarantees.
• Experiments are small (1697 synthetic samples, GPT2-large, single domain) and rely on an LLM judge.
• The sqrt(alpha) vs Omega(alpha) gap and the c(epsilon) separation tightness are conjectured rather than proven.

Experimental setup

Datasets: finance_preference finance_sft

Benchmarks: finance_preference test split

Metrics: win rate estimation error suboptimality bound

Baselines: rDPO DPO SFT

Builds on: DPO RLHF rDPO randomized response

Closest papers by meaning

• Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment (ICML 2025 · similarity 0.90)
• Differentially Private Sparse Reward Estimation with Preference Feedback (NeurIPS 2026 · similarity 0.88)
• Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking (ICLR 2025 · similarity 0.86)
• Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.85)
• Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning (ICLR 2025 · similarity 0.85)
• It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs (NeurIPS 2026 · similarity 0.85)