Research map / Large language models / Alignment and preferences
Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that process reward models trained only on easy MATH problems (levels 1-3) can rerank and guide RL for policy models on hard problems (levels 4-5), enabling easy-to-hard generalization without hard-task human supervision.
Problem
Whether human supervision limited to easy tasks can still enable strong performance on hard reasoning tasks beyond human supervision, i.e., easy-to-hard generalization of generators via evaluators.
Approach
Train outcome and process reward models on easy problems and use them as evaluators to re-rank or RL-train policies on hard problems, exploiting the observation that evaluation generalizes better than generation; also introduces an Outcome & Process Reward Model (OPRM) trained on mixed ORM and PRM data.
Key results
- Process-supervised 7b RL model (PPO/Iterative DPO trained on easy MetaMath/Math-Shepherd data) reaches 34.0% on MATH500, exceeding the previous RL SoTA of 33.0%.
- A 34b Llemma model with reranking at 1024 samples reaches 52.5% on MATH500, per the abstract.
- On APPS, Code Llama-34b with best-of-16 ORM reranking trained only on introductory problems reaches 13.0% strict accuracy overall versus 7.8% for greedy full-data SFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use weighted voting instead of majority voting | Llemma generators scored by easy-trained evaluators · MATH500, especially levels 4-5 | accuracy | Helped |
| Combine outcome and process reward training in an OPRM | Llemma reward models · MATH500 (PRM800K) | reranking accuracy | Helped |
| Add ORM training examples to a PRM on Math-Shepherd (ablation) | Llemma reward models · MATH500 (MetaMath/Math-Shepherd) | reranking accuracy | No clear effect |
| Use easy-task PRM rewards with PPO on all questions instead of final-answer rewards (ablation) | Llemma-7b · MATH500 hard (PRM800K) | accuracy: 10.7 -> 14.9 | Helped |
| Use easy-task PRM rewards with PPO on all questions instead of final-answer rewards (ablation) | Llemma-7b · MATH500 all (PRM800K) | accuracy: 25.6 -> 29.4 | Helped |
| Use easy-task PRM rewards with iterative DPO on easy questions rather than all questions (ablation) | Llemma-7b · MATH500 all (MetaMath/Math-Shepherd) | accuracy: 29.2 -> 34.0 | Helped |
| Train with final-answer labels on hard tasks in addition to easy-trained PRM rewards (ablation) | Llemma-7b PPO · MATH500 hard (MetaMath/Math-Shepherd) | accuracy: 16.0 -> 13.4 | Hurt |
| Use PPO instead of ReST-EM with easy-task rewards | Llemma-7b · MATH500 all (PRM800K) | accuracy: 22.4 -> 27.4 | Helped |
| Use easy-trained ORM reranking with Best-of-16 instead of greedy decoding | Code Llama-34b · APPS All | Average Accuracy (%): 20.1 -> 26.6 | Helped |
| Train SFT on easy tasks only rather than all difficulty levels (ablation) | Llemma-7b · MATH500 (MetaMath), all | accuracy: 31.4 -> 30.0 | Hurt |
| Train SFT on hard tasks only rather than all difficulty levels (ablation) | Llemma-7b · MATH500 (MetaMath), hard | accuracy: 13.7 -> 15.3 | Helped |
| Train SFT on hard tasks only rather than all difficulty levels (ablation) | Llemma-7b · MATH500 (MetaMath), all | accuracy: 31.4 -> 30.4 | Hurt |
Surprising findings
- PPO trained with PRM rewards on only easy (level 1-3) data outperforms SFT and Final-Answer RL trained on the full dataset.
- Hard-only training underperforms easy or all-data training, indicating generalization is limited in the hard-to-easy direction too.
Limitations
- Limited to specific model sizes (7b and 34b Llemma) and mathematical and coding domains.
- Relies on high-quality human supervision on easy tasks and assumes the simulated human demonstrations are reliable.
- Weak base model relative to the hard task limits reward model effectiveness, leading to over-optimization in best-of-n.
Experimental setup
Datasets: MATH PRM800K MetaMath Math-Shepherd APPS
Benchmarks: MATH500 APPS
Metrics: accuracy greedy decoding accuracy maj@16 maj@256 best-of-n accuracy weighted voting accuracy strict accuracy average accuracy
Baselines: Full ICL Hard ICL Easy-to-Hard ICL Full SFT ReST-EM Iterative DPO Final-Answer RL majority voting
Builds on: PPO DPO ReST Llemma Math-Shepherd
Closest papers by meaning
- Debate Helps Weak-to-Strong Generalization (AAAI 2025 · similarity 0.89)
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (ICML 2024 · similarity 0.88)
- Improving Generalization of Alignment with Human Preferences through Group Invariant Learning (ICLR 2024 · similarity 0.87)
- Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization (AAAI 2025 · similarity 0.86)
- The Alignment Problem from a Deep Learning Perspective (ICLR 2024 · similarity 0.86)
- DyCon: Dynamic Reasoning Control via Evolving Difficulty Modeling (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.