Research map / Reinforcement learning / RL from feedback
Hista and Numca: Estimate State Value Effectively for Large Language Model Reinforcement Learning
Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry YANG, Yongqiang Chen, Zhitang Chen, James Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train a PPO critic instead of using group-average baseline (evaluated on unseen data, PPO-1) | PPO · SVEB-Number | MAE: 0.164 -> 0.161 | No clear effect |
| train PPO critic for another epoch (PPO-N) (ablation) | PPO · SVEB-Number | MAE: 0.164 -> 0.152 | Improved |
| replace group-average baseline with Hista hidden-state weighted estimate | GRPO baseline · SVEB-Math | MAE: 0.208 -> 0.145 | Improved |
| use Numca numerical milestones instead of group-average | GRPO baseline · SVEB-Number | MAE: 0.175 -> 0.132 | Improved |
| use Numca on non-numeric domains | GRPO baseline · SVEB-Science | MAE: 0.215 -> 0.217 | Hurt |
| add Numca to GRPO training | GRPO · Math benchmark average | accuracy: 0.541 -> 0.555 | Improved |
| add Hista to DAPO | DAPO · Qwen2.5 base 0.5B, Simple Math Dataset | average accuracy: 19.6 -> 21.2 | Improved |
| add Hista to DAPO | DAPO · Qwen2.5-14B base, Simple Math Dataset | average accuracy: 57.2 -> 58.3 | Improved |
| add Hista to GSPO | GSPO · Qwen2.5-1.5B base, Simple Math Dataset | average accuracy: 37.9 -> 37.7 | No clear effect |
| add Hista to CSIPO | CSIPO · Qwen2.5-1.5B base, Simple Math Dataset | average accuracy: 37.8 -> 38.7 | Improved |
| add Hista to DAPO | DAPO · Hybrid Reasoning Dataset, Qwen2.5-7B instruct, ALL average | average accuracy: 55.2 -> 56.3 | Improved |
| add Hista to DAPO | DAPO · Hard Math Dataset, R1-Distill-1.5B | average accuracy: 60.3 -> 61.4 | Improved |
Closest papers by meaning
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.90)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · similarity 0.88)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.88)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.88)
- LLM-Empowered State Representation for Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
