Research map / Federated learning and privacy / Differential privacy
Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Square-chiPO, a one-line change to chiPO replacing log-loss with a square (Brier) loss over preference probabilities, giving theoretical guarantees for differentially private and label-corrupted offline direct alignment under BT and general preference models.
Problem
Offline preference alignment with label privacy (local and central DP) and Huber label corruption, with general function approximation, where prior work was limited to linear models or had suboptimal/non-vanishing gaps.
Approach
Replace chiPO's log-loss with a bounded square loss over probabilities (Brier score) on sigmoid of the chi-squared-regularized implicit reward difference; scale labels by c(epsilon) for randomized response unbiasedness, and use exponential mechanism for central DP, yielding optimal rates and a unified least-squares generalization analysis under corruption and privacy.
Key results
- Square-chiPO attains O(1/sqrt(n)) suboptimality under local-DP label privacy with general function approximation, matching the optimal privacy cost c(epsilon)=(e^eps+1)/(e^eps-1).
- Under central DP, Square-chiPO gives a pure epsilon-DP guarantee with privacy cost O(1/sqrt(n eps)) for general function classes, versus prior approximate-DP linear results.
- Under Huber corruption, the suboptimality bound adds sqrt(alpha) term, the first meaningful guarantee for DPO-style methods under Huber corruption with general function approximation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace χPO's log-loss with square loss over probabilities | Offline direct alignment under BT preferences · BT preference with label corruption and/or privacy | suboptimality gap | Helped |
| Use square loss instead of log-loss | Offline direct alignment · Huber-corrupted preference labels | suboptimality gap | Helped |
| Apply corruption before local differential privacy rather than after it | SquareχPO · BT preference, local privacy and Huber corruption | suboptimality gap: √α vs. √(α · c(ε)) | Helped |
| Add central differential privacy to the corrupted-label setting | SquareχPO · BT preference, cDP | suboptimality gap: (1+1/√ε)√(log(|Π|/ζ)/n) + √α | Mixed |
| Use square loss over probabilities rather than log-loss | Preference-label estimation · Huber-corrupted labels | loss boundedness: 1 | Helped |
Surprising findings
- Corruption applied after local privacy (LTC) yields a multiplicative sqrt(c(epsilon)) interaction with corruption, while CTL is additive, showing ordering matters.
Limitations
- Central-DP algorithm requires sampling from the policy class, which is computationally inefficient for infinite classes.
- Experiments are only proof-of-concept and the paper is primarily theoretical.
- Analysis assumes finite policy class and bounded implicit reward differences, with alpha-dependence in generalization bound possibly loose (alpha vs alpha^2).
Experimental setup
Metrics: suboptimality gap duality gap
Baselines: chiPO DPO IPO
Builds on: chiPO DPO IPO Nash-MD / self-play preference optimization
Closest papers by meaning
- A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO (ICML 2025 · similarity 0.90)
- Differentially Private Sparse Reward Estimation with Preference Feedback (NeurIPS 2026 · similarity 0.89)
- Distributionally Robust Listwise Preference Optimization (NeurIPS 2026 · similarity 0.88)
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.87)
- Oracle-Robust Online Alignment for Large Language Models (NeurIPS 2026 · similarity 0.87)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.