Research map / Federated learning and privacy / Differential privacy
Rank-Aware Differentially Private Release of Listwise Preferences for LLM Alignment
Junwei Chen, Manjiang Yu, Pengpeng Qiao, Yang Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a rank-aware local differential privacy mechanism for releasing listwise preference feedback to align LLMs, extending privacy-preserving alignment beyond pairwise comparisons.
Problem
Listwise preference feedback reveals more sensitive user preferences than pairwise comparisons, and privacy-preserving alignment under local differential privacy has mainly addressed pairwise feedback, leaving listwise alignment under LDP unexplored.
Approach
Extends local differential privacy to listwise preference release by making the privatization mechanism rank-aware, so that privacy noise is tailored to the structure of full rankings rather than treating them as independent pairwise comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace randomized response with rank-aware exponential mechanism for private binary release | KTO on privatized listwise labels (Qwen2-0.5B) · ListUltraFeedback, eps=0.5 | average win rate vs SFT: 48.0 -> 51.2 | Improved |
| replace randomized response with rank-aware exponential mechanism | KTO (Qwen2-0.5B) · ListUltraFeedback, eps=3 | average win rate vs SFT: 57.1 -> 59.6 | Improved |
| replace randomized response with rank-aware exponential mechanism | KTO (Mistral-7B) · ListUltraFeedback, eps=0.5 | average win rate vs SFT: 84.0 -> 85.2 | Improved |
| replace randomized response with rank-aware exponential mechanism | KTO (Mistral-7B) · ListUltraFeedback, eps=3 | average win rate vs SFT: 85.8 -> 86.7 | Improved |
| replace randomized response with rank-aware exponential mechanism | KTO (Qwen2-0.5B) · MT-Bench, eps=0.5 | win rate vs SFT: 52.5 -> 50.1 | Hurt |
| replace randomized response with rank-aware exponential mechanism | KTO (Qwen2-0.5B) · AlpacaEval, eps=1 | win rate vs SFT: 50.7 -> 50.6 | No clear effect |
| replace randomized response with rank-aware exponential mechanism | KTO (Qwen2-0.5B) · Scoring (ArmoRM) win rate, eps=1 | win rate vs SFT: 43.7 -> 49.1 | Improved |
| release binary top-B labels instead of cardinal scores or full ranks to cut sensitivity | private release of listwise preferences · sensitivity analysis (m items) | sensitivity / noise scale: m^2 -> 2(m-1) -> 2 | Improved |
| use linear boundary score rather than other monotone rank scores (ablation) | rank-aware utility · calibrated rank span (theory) | S_w/Delta_w | Improved |
| add privatization at private privacy budget epsilon compared with non-private training | KTO (Qwen2-0.5B) · ListUltraFeedback average, eps=3 vs non-private | average win rate vs SFT: 60.8 -> 59.6 | Hurt |
| keep full candidate list for private release instead of random small subset (ablation) | RAEM+KTO (Qwen2-0.5B) · ListUltraFeedback, eps=1 | average proxy win rate | Improved |
| increase list size m under oracle top-bottom selection (ablation) | RAEM+KTO (Qwen2-0.5B) · ListUltraFeedback, eps=1 | average proxy win rate | No clear effect |
Closest papers by meaning
- Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment (ICML 2025 · similarity 0.89)
- Distributionally Robust Listwise Preference Optimization (NeurIPS 2026 · similarity 0.89)
- Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.89)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.87)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.87)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
