Research map / Reinforcement learning / Policy gradient and actor-critic
$V_0$: A Generalist Value Model for Any Policy at State Zero
Yi-Kai Zhang, Zhiyuan Yao, Hongyan Hao, Yueqing Sun, Qi GU, Hui Su, Xunliang Cai, De-Chuan Zhan, Han-Jia Ye
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add in-context policy capability history (V0) instead of coupled value model | value model · GRPO tracking, DeepSeek-R1-Distill-Qwen-1.5B | Intra AUC (1st epoch): 0.708 -> 0.887 | Improved |
| V0 vs Vanilla VM on unseen query IDs | value model · Strict generalization, 1.5B | Intra AUC: 0.560 -> 0.710 | Improved |
| use V0-based budget allocation in GRPO | Qwen3-4B-Instruct-2507 GRPO · OlympiadBench Mean@16 | accuracy: ~2% increase | Improved |
| use V0 for inference routing | model fleet routing · 12 benchmarks | accuracy-cost Pareto | Improved |
| use Residual Dynamic Query connector instead of last_token (ablation) | V0 · connector ablation | Intra AUC: 0.621 -> 0.705 | Improved |
| use Residual Dynamic Query instead of Fixed Query (ablation) | V0 · connector ablation | Intra AUC: 0.674 -> 0.705 | Improved |
| use MLP/Cascaded/MultiScale connectors (ablation) | V0 · connector ablation | Intra AUC: 0.618 / 0.585 / 0.589 vs 0.705 | Hurt |
| add soft CE to pairwise ranking loss (ablation) | V0 · loss ablation | Intra AUC: 0.578 -> 0.705 | Improved |
| add pairwise ranking loss to soft CE (ablation) | V0 · loss ablation | Intra AUC: 0.686 -> 0.705 | Improved |
| add pairwise ranking loss to soft CE (ablation) | V0 · loss ablation | Pairwise accuracy: 0.783 -> 0.839 | Improved |
| tune TabPFN head jointly with connector (ablation) | V0 · tuning ablation | Intra AUC: 0.705 -> 0.594 | Hurt |
| add connector over last_token (frozen TabPFN) (ablation) | V0 · tuning ablation | Intra AUC: 0.621 -> 0.705 | Improved |
Closest papers by meaning
- Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor’s Internal States (NeurIPS 2026 · similarity 0.89)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.88)
- Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex (NeurIPS 2026 · similarity 0.87)
- Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.87)
- Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.86)
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
