Research map / Learning theory and algorithms / Combinatorial and search optimization
Meta-Learning Objectives for Preference Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a MuJoCo diagnostic suite for preference optimization, a mirror-descent family (MPO) whose objectives are meta-learned with evolution strategies, and a temporally-aware variant (TeMPO) that transfers to LLM alignment.
Problem
Comparing preference optimization (PO) algorithms for LLM alignment is costly, noisy, and confounded by model size and hyperparameters; existing benchmarks do not cleanly reveal algorithm failure modes under noisy or mixed-quality preference data.
Approach
Use MuJoCo (Ant/Three-legged-ant, Hopper) with ground-truth rewards as a controlled PO benchmark, define a mirror-descent-based family of PO objectives (MPO) with a neural parametrization of its link and potential functions, meta-learn them with OpenAI-ES, and derive TeMPO, which gradually shifts from SFT to a non-truncating PO loss, based on insights from the discovered objectives.
Key results
- On TLA base setting, discovered 1S-MPO (noisy, eps=0.3) with parametrized g reaches 3931 ± 69 and 3417 ± 82 at eps=0.3, versus best human-designed baseline SimPO 3683 on base and SFT 2049 at eps=0.3.
- On AlpacaEval with gemma-7b and dpo-mix-7k, TeMPO (3) achieves 35.4% win-rate (length-controlled 29.1%), above DPO's 30.7% and SimPO's 32.5%; on mistral-7b it reaches 41.6% vs DPO 32.8%.
- Ablation on gemma-7b shows SFT+SimPO with gamma=1 reaches 35.07% on capybara-7k versus 27.52% for SimPO with gamma=0 (win-rate).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the discovered 1S-MPO objective for each dataset setting | Preference optimization on TLA · TLA Base, Mixed Quality, Noisy (ε=0.1), and Noisy (ε=0.3) | reward: 3683 $\pm$78 -> 3931 $\pm$ 69; 3117 $\pm$ 185 -> 3627 $\pm$ 79; 3162 $\pm$ 66 -> 3834 $\pm$ 82; 2049 $\pm$33 -> 3417 $\p | Helped |
| Use TeMPO with the sigmoid schedule α(t)=σ(20(x−0.75)) | TeMPO · AlpacaEval, gemma-7b + dpo-mix-7k | win-rate: 30.7$\pm$1.6 -> 35.4$\pm$1.7 | Helped |
| Use TeMPO with the sigmoid schedule α(t)=σ(20(x−0.75)) | TeMPO · AlpacaEval, gemma-7b + capybara-7k | win-rate: 37.9$\pm$1.7 -> 39.4$\pm$1.7 | Helped |
| Use TeMPO with the sigmoid schedule α(t)=σ(20(x−0.75)) | TeMPO · AlpacaEval, mistral-7b + dpo-mix-7k | win-rate: 32.8$\pm$1.6 -> 41.6$\pm$1.7 | Helped |
| Use TeMPO with the sigmoid schedule α(t)=σ(20(x−0.75)) | TeMPO · AlpacaEval, gemma-7b + capybara-7k | length-controlled win-rate: 32.8$\pm$0.2 -> 33.8$\pm$0.1 | Helped |
| Use TeMPO with the sigmoid schedule α(t)=σ(20(x−0.75)) | TeMPO · AlpacaEval, mistral-7b + dpo-mix-7k | length-controlled win-rate: 29.5$\pm$0.3 -> 30.6$\pm$0.2 | Helped |
| Increase SimPO γ from 0 to 10 (ablation) | SimPO on gemma-7b · AlpacaEval, dpo-mix-7k | win-rate: 30.50 -> 33.60 | Helped |
| Increase SimPO γ from 0 to 10 (ablation) | SimPO on gemma-7b · AlpacaEval, capybara-7k | win-rate: 27.52 -> 27.39 | No clear effect |
| Add an SFT term to SimPO (γ=1) (ablation) | SimPO on gemma-7b · AlpacaEval, capybara-7k | win-rate: 27.52 -> 35.07 | Helped |
| Continue optimizing after the chosen trajectory is more probable than the rejected trajectory | Preference optimization objectives · TLA | reward: 3255 $\pm$ 66 -> 3774 $\pm$ 102 | Helped |
| Use temporal awareness to gradually switch from SFT to preference optimization (ablation) | TeMPO · AlpacaEval LLM alignment | win-rate | Helped |
Surprising findings
- Algorithms with discovered parametrized g performed poorly when transferred as static objectives to LLM alignment, which the authors attribute to overfitting to the TLA task's dense state-action coverage.
- SimPO, without any SFT term, performs catastrophically on high-noise settings (-3828 at eps=0.3) despite being top on base data.
Limitations
- LLM transfer is tested only on two 7B base models and two preference datasets with AlpacaEval, and the discovered static objectives did not transfer well.
- Discovered objectives were searched per dataset setting on a single MuJoCo task family, so generalization beyond TLA and Hopper is not established.
- Offline setting only; online/on-policy PO methods are not evaluated.
Experimental setup
Datasets: dpo-mix-7k distilabel-capybara-dpo-7k-binarized MuJoCo TLA preference datasets (generated) MuJoCo Hopper preference datasets (generated)
Benchmarks: Three-legged-ant (TLA) MuJoCo Hopper MuJoCo AlpacaEval
Metrics: average cumulative reward win-rate length-controlled win-rate
Baselines: RRHF SLiC-HF DPO IPO CPO ORPO R-DPO SimPO SFT KTO f-DPO (Jensen-Shannon)
Builds on: DPO ORPO SimPO OpenAI-ES Alignment Handbook
Closest papers by meaning
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.90)
- RePO: Understanding Preference Learning Through ReLU-Based Optimization (NeurIPS 2025 · similarity 0.89)
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.88)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.88)
- Generalized Preference Optimization: A Unified Approach to Offline Alignment (ICML 2024 · similarity 0.88)
- Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.