Research map / Large language models / Alignment and preferences

Weighted-Reward Preference Optimization for Implicit Model Fusion

ICLR 2025 · Large language models › Alignment and preferences · approach: Robust and calibrated DPO · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Weighted-Reward Preference Optimization (WRPO), an implicit model-fusion method that fuses heterogeneous open-source LLMs into LLaMA3-8B-Instruct via preference optimization with a progressively increasing weight on source-model preferred responses, avoiding vocabulary alignment and distribution merging.

Problem

Fusing heterogeneous open-source LLMs of different architectures and sizes into one target model is hard because explicit fusion requires vocabulary alignment and distribution merging, and naive preference optimization with off-policy source responses suffers from distribution shift.

Approach

Implicit fusion via preference optimization: the preferred response is a weighted mix of the best source-LLM response and the best target-LLM response, with the source weight alpha linearly increased from 0 to a target value during training, so the target gradually shifts its distribution toward source LLMs.

Key results

• WRPO on LLaMA3-8B-Instruct reaches 55.9% length-controlled win rate on AlpacaEval-2 and 46.2% win rate on Arena-Hard, outperforming the Target-DPO baseline (48.2 LC) and FuseChat (38.1 LC).
• Target-SFT-WRPO-Medium (5 source LLMs) beats FuseLLM and FuseChat by 17.5 and 15.4 LC win-rate points on AlpacaEval-2 and beats them by 9.5 and 8.9 WR points on Arena-Hard.
• Starting from Target-SFT, WRPO improves over DPO by 5.2 points LC win rate on AlpacaEval-2 and 6.0 points win rate on Arena-Hard.

Measured effects reported in the paper

Surprising findings

• Naively applying DPO on preference pairs with source-model responses yields suboptimal results compared with on-policy target-only preference data.
• Including source-model dispreferred responses degraded AlpacaEval-2 and MT-Bench performance, because source-model rejected responses often scored higher than the target's preferred responses.

Limitations

• Only the single highest-reward source response is used as preferred output per prompt, potentially underusing source capabilities.
• Relies heavily on existing preference optimization frameworks, lacking rigorous theoretical analysis.
• Limited gains on knowledge-heavy tasks such as MMLU due to the narrow instruction-following training dataset.

Experimental setup

Datasets: UltraFeedback Arena-Hard MT-Bench AlpacaEval-2 Open LLM Leaderboard tasks

Benchmarks: AlpacaEval-2 Arena-Hard MT-Bench Open LLM Leaderboard (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K)

Metrics: length-controlled win rate win rate MT-Bench score accuracy

Baselines: PackLLM-Top1-PPL LLM-Blender-Top1 MoA FuseLLM FuseChat DPO SimPO IPO

Builds on: DPO SimPO IPO FuseLLM FuseChat WPO

Closest papers by meaning

• InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration (NeurIPS 2025 · similarity 0.88)
• Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.86)
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning (NeurIPS 2025 · similarity 0.86)
• Diverse Preference Learning for Capabilities and Alignment (ICLR 2025 · similarity 0.86)
• SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.86)