Research map / Federated learning and privacy / Differential privacy

Differentially Private Sparse Reward Estimation with Preference Feedback

NeurIPS 2026 · Federated learning and privacy › Differential privacy · approach: Private online and user-level learning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies reward estimation from pairwise BTL preference feedback under label differential privacy, giving minimax upper and lower bounds in local (randomized response) and central (objective perturbation) models.

Problem

Estimating a linear BTL reward parameter from pairwise human preference labels while protecting labelers' privacy under label differential privacy (local and central models).

Approach

Applies label-DP to BTL reward estimation; for the local model it uses randomized response with a de-biased loss that keeps log-odds unbiased, and for the central model it uses objective perturbation with Gaussian noise, with matching lower bounds on estimation error.

Key results

• Local label-DP cost is Theta(1/(e^eps-1) * sqrt(d/n)) for semi-norm error, with matching upper bound via randomized response with de-biased loss.
• Central label-DP cost is an additive lower-order term of order 1/(eps n), with objective perturbation matching lower bound up to a factor d in the best case.
• Simulations with d=5 show non-private MLE has lowest error, then central objective perturbation, then local RR.

Surprising findings

• Under the local model, privacy cost is of the same order as the statistical error term, while under the central model it is only a lower-order additive term.

Limitations

• Upper bounds under local model have an exp(LB) factor gap from lower bounds
• Central model upper bound has a gap of order d versus lower bound
• Only linear reward model and synthetic experiments; no LLM experiments

Experimental setup

Datasets: synthetic BTL data

Benchmarks: synthetic BTL simulations

Metrics: l2 estimation error

Baselines: MLE (non-private) Randomized response MLE

Builds on: Zhu et al. 2023 linear BTL reward estimation Shah et al. 2015 BTL estimation Ghazi et al. 2021 label DP Kifer et al. objective perturbation

Closest papers by meaning

• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
• Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.89)
• Square$\chi$PO: Differentially Private and Robust $\chi^2$-Preference Optimization in Offline Direct Alignment (ICML 2025 · similarity 0.89)
• PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.88)
• Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (ICLR 2024 · similarity 0.88)
• MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.88)