Research map / Large language models / Alignment and preferences

Towards Federated RLHF with Aggregated Client Preference for LLMs

ICLR 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes federated RLHF in which clients train binary preference selectors via FedAvg-style aggregation, and FedBiscuit extends this with cluster-wise selector aggregation and majority voting to counter preference heterogeneity and reward hacking, then uses the selectors to build DPO data.

Problem

RLHF needs user preference data, but users are reluctant to share private preference data centrally; federated preference collection suffers from computation overhead, preference heterogeneity across clients, and reward hacking.

Approach

Replace per-client reward models with binary selectors trained by cross-entropy in FL, aggregate them into a global selector used to generate DPO preference pairs; the extended method clusters clients by preference loss, trains U selectors per cluster, and labels pairs by majority vote across selectors to resist reward hacking.

Key results

• On summarization, the Qwen-2-0.5B-selector FedBis (single selector) reaches 86.69% win rate on Gemma-2B and 89.63% on LLaMA-2-7B versus 67.27% and 76.79% for the raw models, while FedAvg drops to about 28%.
• On AlpacaEval 2.0, FedBis achieves up to 4.58% LC win rate for Gemma-2B and FedBiscuit (U=3) reaches 5.63% for LLaMA-2-7B under SafeRLHF prompts, versus 2.40% and 4.80% for raw models.
• FedBiscuit (U=3) with Gemma selector gives 83.81% and 91.32% summarization win rates on Gemma-2B and LLaMA-2-7B respectively, the best under that selector.

Measured effects reported in the paper

Surprising findings

• FedAvg, which aggregates fine-tuned LLMs directly, drops win rates by about 20-50 points below the raw model on summarization.
• The single-selector FedBis outperforms the multi-selector FedBiscuit on summarization, which the authors attribute to low preference heterogeneity in that task.

Limitations

• FedBiscuit adds extra communication cost from periodic client regrouping, scaling with the number of selectors and regrouping interval.
• Improvements are small on AlpacaEval 2.0 (length-controlled win rates around 4-6%), and the best method varies by task and selector.
• Evaluation relies on automatic judges (Auto-J, GPT-4-turbo) without human evaluation.

Experimental setup

Datasets: TL;DR comparison OpenAI TL;DR SHP SafeRLHF Alpaca

Benchmarks: TL;DR summarization test (6,553 samples, Auto-J win rate) AlpacaEval 2.0 (805 instructions)

Metrics: win rate length-controlled win rate

Baselines: FedAvg FedDPO Raw Model

Builds on: FedAvg DPO OpenFedLLM FederatedScope LoRA

Closest papers by meaning

• Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.88)
• MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.88)
• Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (ICLR 2024 · similarity 0.88)
• Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.87)
• Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.87)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.87)