Research map / Large language models / Alignment and preferences
Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WSPO, which aligns a strong LM by matching its log-probability ratio to the change a weak model undergoes from alignment (before vs after DPO/PPO), transferring and amplifying weak-model alignment.
Problem
Aligning LMs with human preferences typically needs preference pairs and reward models or DPO-style training; the paper asks whether the alignment signal from a weak aligned model can be used to align a stronger model (weak-to-strong alignment).
Approach
Derive from the KL-constrained RL optimal policy that a reward equals the log-ratio between aligned and reference policies; use the weak model's log-ratio (aligned vs reference) as a target for the strong model's log-ratio via a squared-loss objective on offline data, with a single hyperparameter gamma, so no paired preference data or explicit reward model is needed.
Key results
- On Qwen2-7B-Instruct, WSPO raises Arena-Hard win rate from 39.70 (SFT) to 49.60 and AlpacaEval 2 LC win rate to 47.04, with MT-Bench 7.33 (vs 7.19 SFT, 7.26 DPO).
- On Llama3.1-8B-Instruct aligned by Llama3.2-1B weak model, WSPO reaches Arena-Hard 57.20 vs 48.30 SFT and 48.20 DPO.
- On the single-turn Anthropic-HH dialogue task (Qwen2-7B, GPT-4o-mini judge), WSPO win rate 61.83 vs DPO 57.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer alignment from Qwen2-1.5B-Instruct using WSPO instead of leaving the Qwen2-7B-Instruct model at SFT | Qwen2-7B-Instruct · Arena-Hard | win rate (%): 39.70 -> 49.60 | Helped |
| Transfer alignment from Qwen2-1.5B-Base using WSPO rather than DPO | Qwen2-7B-Base · Arena-Hard | win rate (%): 10.70 -> 29.00 | Helped |
| Transfer alignment from Qwen2-1.5B-Instruct using WSPO rather than DPO | Qwen2-7B-Instruct · AlpacaEval 2 | length-controlled win rate (%): 32.10 -> 47.04 | Helped |
| Apply WSPO alignment | Qwen2-7B-Instruct · TruthfulQA | score: 41.00 -> 47.00 | Helped |
| Apply WSPO alignment | Qwen2-7B-Instruct · GSM8K | score: 77.86 -> 77.94 | No clear effect |
| Use DPO alignment on UltraFeedback | Qwen2-7B-Instruct · Arena-Hard | win rate (%): 39.70 -> 39.30 | Hurt |
| Use gamma 0.1 rather than gamma 1.0 in WSPO (ablation) | Qwen2-7B-Base with same-size weak-model checkpoints · Arena-Hard | win rate: 10.90 -> 15.30 | Helped |
| Use gamma 0.5 rather than gamma 1.0 in WSPO (ablation) | Qwen2-7B-Base with same-size weak-model checkpoints · Arena-Hard | win rate: 10.90 -> 14.90 | Helped |
| Use a 0.5B weak model without DPO alignment enhancement (ablation) | Qwen2-7B-Instruct WSPO · Arena-Hard | score: 45.00 | Mixed |
| Train WSPO on the rejected preference subset rather than preferred data (ablation) | Qwen2-7B-Instruct · Arena-Hard | score: 34.40 -> 40.30 | Helped |
| Use Qwen2-7B SFT and DPO checkpoints as the weak-model pair (ablation) | Qwen2-7B-Base WSPO · Arena-Hard | score: 11.20 -> 15.30 | Helped |
| Use WSPO rather than DPO to align Llama3.1-8B-Instruct from Llama3.2-1B-Instruct | Llama3.1-8B-Instruct · Arena-Hard | win rate (%): 48.20 -> 57.20 | Helped |
Surprising findings
- Aligning from the rejected subset of the preference dataset still yields gains (Qwen2-7B-Instruct Arena-Hard 40.30 vs SFT 34.40 under Qwen2.5-72B judge).
- DPO on UltraFeedback slightly degrades Qwen2-7B-Instruct performance, while WSPO improves it.
Limitations
- Does not explore transfer across different model architectures or the effect of weak-model alignment strength.
- Does not explain why alignment transfer amplifies rather than merely copies the weak model's behavior.
- Requires loading two weak models plus the strong model, increasing memory and compute; the only tuned hyperparameter is gamma, but DPO baselines were hard to tune.
Experimental setup
Datasets: XSUM Anthropic HH UltraChat-200k UltraFeedback RLHF-V
Benchmarks: AlpacaEval 2 Arena-Hard v0.1 MT-Bench MMLU CMMLU TruthfulQA GSM-PLUS GSM8K MMHal-Bench
Metrics: length-controlled win rate win rate MT-Bench score accuracy hallucination rate informativeness
Baselines: SFT DPO PPO SimPO Preferred-FT
Builds on: DPO PPO weak-to-strong generalization KL-constrained RLHF reparameterization
Closest papers by meaning
- Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.92)
- MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization (ICLR 2025 · similarity 0.91)
- Bayesian WeakS-to-Strong from Text Classification to Generation (ICLR 2025 · similarity 0.91)
- RePO: Understanding Preference Learning Through ReLU-Based Optimization (NeurIPS 2025 · similarity 0.90)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.90)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.