Research map / Large language models / Alignment and preferences

Online Preference Alignment for Language Models via Count-based Exploration

ICLR 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes COPO, an online RLHF method that adds a count-based exploration bonus (pseudo-counts from a coin-flipping network) to the DPO objective, with a regret analysis under linear rewards, and evaluates on Zephyr-7B and Llama-3-8B-Instruct.

Problem

Online RLHF for LLMs needs systematic exploration of the prompt-response space, since a fixed preference dataset has limited coverage and the learned reward generalizes poorly to out-of-distribution responses.

Approach

Derives an optimistic (UCB) RLHF objective whose exploration term reduces to a count-based bonus, and combines DPO with that bonus, estimating pseudo-counts for prompt-response pairs with a lightweight coin-flipping network trained on the frozen LLM's last hidden state.

Key results

• COPO raises AlpacaEval 2.0 LC win rate from 22.19 to 27.21 on Zephyr-7B (iter 2) and from 33.17 to 35.54 on Llama-3-8B-Instruct (iter 3).
• On Llama-3-8B-Instruct, COPO iter 3 (35.54 LC win rate) outperforms Yi-34B-Chat (27.19) and Llama-3-70B-Instruct (33.17) in LC win rate.
• With the 20% preference-data subset, COPO iter3 reaches 31.80 LC win rate versus 29.25 for SELM and 28.16 for online DPO.

Measured effects reported in the paper

Surprising findings

• On Zephyr-7B, response length grows substantially in later iterations, lowering LC win rate, which the authors attribute to the exploration term favoring novel (longer) responses.
• Using a stronger 8B reward model (ArmoRM) lifts COPO to 41.78 LC win rate, versus 35.54 with PairRM-0.4B.

Limitations

• Exploration term can increase response length, reducing LC win rate in later Zephyr iterations.
• Theoretical analysis holds under linear reward and tabular assumptions; the practical CFN pseudo-count is an approximation.
• Uses a fixed prompt set from the initial dataset, limiting exploration coverage; results depend on the reward model quality.

Experimental setup

Datasets: UltraFeedback 60K (HuggingFaceH4/ultrafeedback_binarized)

Benchmarks: AlpacaEval 2.0 MT-Bench GSM8K HellaSwag ARC TruthfulQA EQ-Bench OpenBookQA

Metrics: LC win rate win rate average MT-Bench score accuracy

Baselines: Online DPO SELM SPIN SPPO DNO Yi-34B-Chat Llama-3-70B-Instruct

Builds on: DPO SELM count-based exploration (CFN) Online DPO

Closest papers by meaning

• Avoiding exp(R) scaling in RLHF through Preference-based Exploration (NeurIPS 2025 · similarity 0.96)
• Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.93)
• Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.93)
• Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.92)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
• Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.92)