Research map / Large language models / Alignment and preferences
InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfiFPO, a preference optimization method that replaces the DPO reference model with a sequence-level fused distribution of multiple source LLMs, improving Phi-4 from 79.95 to 83.33 average over 11 benchmarks.
Problem
Model fusion has focused on SFT; fusing knowledge during preference alignment is underexplored, and existing fusion methods (token-level KL, WRPO) face vocabulary mismatch or discard source probabilities and negative signals.
Approach
Replace the DPO reference model with a geometric-mean fused source distribution computed at the sequence level (implicit model fusion), with length normalization, probability clipping, and max-margin source selection, derived offline from a sequence-KL constrained RLHF objective.
Key results
- InfiFPO raises Phi-4 average over 11 benchmarks from 79.95 to 83.33, outperforming SFT-WRPO (82.80) and InfiFusion (82.38).
- InfiFPO* uses 55 GPU hours versus 160 for InfiFusion* while scoring higher on average (82.74 vs 82.38).
- Ablation: removing both length normalization and probability clipping drops InfiFPO to 79.51, below the Phi-4 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply InfiFPO preference optimization and source-model fusion | Phi-4 · 11 benchmarks | average score: 79.95 -> 83.33 | Helped |
| replace InfiFusion* with InfiFPO* | Phi-4 · 11 benchmarks | average score: +0.36 | Helped |
| replace SFT-WRPO with InfiFPO | Phi-4 · 11 benchmarks | average score: +0.53 | Helped |
| add InfiFPO fusion to WRPO | WRPO · Math | score: +0.4 | Helped |
| add InfiFPO fusion to WRPO | WRPO · Code | score: +0.2 | Helped |
| add InfiFPO fusion to IPO | IPO · All benchmarks | average score: +0.8 | Helped |
| remove length normalization and probability clipping (ablation) | InfiFPO · All benchmarks | average score: 83.33 -> 79.51 | Hurt |
| add length normalization (ablation) | InfiFPO · All benchmarks | average score: +2.9 | Helped |
| add probability clipping (ablation) | InfiFPO · All benchmarks | average score: +1.5 | Helped |
| add both length normalization and probability clipping (ablation) | InfiFPO · All benchmarks | average score: +3.3 | Helped |
| increase source models from 1 to 5 (ablation) | InfiFPO · All benchmarks | average score: 81.54 -> 83.33 | Helped |
| remove sequence alignment (ablation) | InfiFPO · All benchmarks | average score: 82.74 -> 80.82 | Hurt |
Surprising findings
- Without length normalization and probability clipping, InfiFPO underperforms the pivot model Phi-4 (79.51 vs 79.95).
- Token-level fusion methods such as FuseChat without sequence alignment perform worse than the base Phi-4 model.
Limitations
- Relies on existing preference optimization objectives such as DPO and lacks rigorous theoretical analysis of fusion mechanisms.
- Only five mid-sized open-source source models were tested, not SOTA models.
- Experiments with larger-scale models and datasets remain unexplored.
Experimental setup
Datasets: Infinity-Instruct NuminaMath-1.5 KodCode-V1-SFT UltraFeedback
Benchmarks: GSM8K MATH TheoremQA MBPP HumanEval BBH ARC-C MMLU IFEval DROP HellaSwag
Metrics: accuracy average score GPU hours
Baselines: FuseLLM FuseChat InfiFusion SFT SFT-DPO SFT-IPO SFT-WRPO
Builds on: Direct Preference Optimization FuseLLM WRPO IPO InfiFusion
Closest papers by meaning
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.91)
- Multi-Reference Preference Optimization for Large Language Models (AAAI 2025 · similarity 0.91)
- Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models (ICML 2025 · similarity 0.91)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.91)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
- Model Fusion through Bayesian Optimization in Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.