Research map / Reinforcement learning / RL from feedback
Hybrid Reinforcement: when reward is sparse, better to be dense
Leitian Tao, Ilia Kulikov, Swarnadeep Saha, Tianlu Wang, Jing Xu, Sharon Li, Jason E Weston, Ping Yu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stratified hybrid reward (HERO) vs RM-only | Qwen3-4B-Base · hard-to-verify eval, trained on easy | avg accuracy: 54.6 -> 66.3 | Improved |
| stratified hybrid reward (HERO) vs verifier-only | Qwen3-4B-Base · hard-to-verify eval, trained on easy | avg accuracy: 57.1 -> 66.3 | Improved |
| HERO vs verifier-only on hard-to-verify training data | Qwen3-4B-Base · easy-to-verify eval, trained on hard | avg pass@1: 47.4 -> 56.8 | Improved |
| verifier-only training on hard-to-verify data vs SFT cold start | Qwen3-4B-Base · hard-to-verify avg | avg accuracy: 47.1 -> 47.4 | Hurt |
| HERO vs baselines | OctoThinker-8B-Hybrid-Base · easy and hard avg, trained on hard | avg pass@1: 34.6 -> 41.0 | Improved |
| replace naive reward combination with stratified normalization (ablation) | HERO · Easy-to-verify avg | avg pass@1: 58.7 -> 62.0 | Improved |
| replace naive reward combination with stratified normalization (ablation) | HERO · HardVerify-Math | accuracy: 61.4 -> 73.2 | Improved |
| add variance-aware reweighting (ablation) | HERO · HardVerify-Math | accuracy: 69.4 -> 73.2 | Improved |
| add variance-aware reweighting (ablation) | HERO · Easy-to-verify avg | avg pass@1: 60.8 -> 62.0 | Improved |
| scale reward model from 7B to 72B (ablation) | HERO · Easy-to-verify avg | avg pass@1: 62.0 -> 62.8 | No clear effect |
| scale reward model from 7B to 72B (ablation) | HERO · HardVerify-Math | accuracy: 73.2 -> 71.4 | Hurt |
| use dense reward on negative group (keep sparse positives) (ablation) | HERO · verifiable and hard-to-verify tasks | accuracy: 59.4 -> 61.4 (verifiable); 62.2 -> 68.4 (hard) | Improved |
Closest papers by meaning
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.89)
- A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.89)
- Learning to Reason without External Rewards (ICLR 2026 · similarity 0.89)
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.89)
- VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models (ICLR 2026 · similarity 0.88)
- Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
