Research map / Multi-agent systems and games / LLM agents
Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
Wenxuan Zhang, Yuhui Wang, Donggang Jia, Xiaoqian Shen, Jian Ding, Ivan Viola, Jürgen Schmidhuber, Mohamed Elhoseiny
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add turn-level advantage to token-level advantage (alpha 0 -> 0.5) (ablation) | HyGAE · Sokoban | success rate: 0.59 -> 0.83 | Improved |
| add token-level advantage to turn-level advantage (alpha 1 -> 0.5) (ablation) | HyGAE · Sokoban | success rate: 0.38 -> 0.83 | Improved |
| mix turn and token advantage (alpha 0 -> 0.5) (ablation) | HyGAE · FrozenLake | success rate: 0.72 -> 0.80 | Improved |
| mix turn and token advantage (alpha 1 -> 0.5) (ablation) | HyGAE · FrozenLake | success rate: 0.70 -> 0.80 | Improved |
| mix turn and token advantage (alpha 0 -> 0.5) (ablation) | HyGAE · Primitive Skill (avg.) | success rate: 0.98 -> 1.00 | Improved |
| mix turn and token advantage (alpha 1 -> 0.5) (ablation) | HyGAE · Primitive Skill (avg.) | success rate: 0.25 -> 1.00 | Improved |
| use unbalanced mixing alpha 0.25 or 0.75 instead of 0.5 (ablation) | HyGAE · Primitive Skill (avg.) | success rate: 1.00 -> 0.73 | Hurt |
| replace PPO with hybrid advantage and unified critic (HyGAE) | PPO · Sokoban | success rate: 0.5797 ± 0.2105 -> 0.8222 ± 0.0711 | Improved |
| use hybrid advantage and unified critic instead of best token-wise baseline | Token-PPO · Average over five benchmarks | success rate: 0.81 -> 0.91 | Improved |
| broadcast turn-level return uniformly across tokens, or drop the value model (REINFORCE), instead of the mixed return (ablation) | HyGAE · Sokoban | training stability | Hurt |
| aggregate small token-level rewards (KL penalties) into the turn reward (ablation) | HyGAE · Sokoban | success rate | No clear effect |
| use turn-level value as constant baseline instead of token-specific value for broadcast turn advantage (ablation) | HyGAE · Sokoban | success rate | No clear effect |
Closest papers by meaning
- VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents (NeurIPS 2025 · similarity 0.88)
- Agentic Multi-Turn Reasoning: A Fairness Approach (NeurIPS 2026 · similarity 0.87)
- Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.87)
- COVR: Collaborative Optimization of VLMs and RL Agent for Visual-Based Control (AAAI 2026 · similarity 0.86)
- A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective (NeurIPS 2024 · similarity 0.86)
- Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
