Research map / Reinforcement learning / RL from feedback
Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction
Haoyu Peter Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate token rewards into turn-wise rewards with softmax normalization (Norm-ITPO) instead of sparse trajectory reward | RLOO, Qwen2.5-3B-Instruct · Math Tutoring | accuracy: 29.06 -> 32.50 | Improved |
| Add Norm-ITPO turn rewards over trajectory-level reward | RLOO, Qwen2.5-3B-Instruct · Medical Recommendation | diagnosis score: 65.12 -> 69.24 | Improved |
| Add Norm-ITPO turn rewards over trajectory-level reward | GRPO, Qwen2.5-3B-Instruct · Medical Recommendation | diagnosis score: 66.41 -> 71.74 | Improved |
| Add Norm-ITPO turn rewards to PPO value model | PPO, Qwen2.5-3B-Instruct · Doc Writing | BLEU: 40.57 -> 45.16 | Improved |
| Add reward normalization (Norm-ITPO vs ITPO) (ablation) | PPO with value model · Medical Recommendation | overall score: 54.87 -> 59.44 | Improved |
| Add reward normalization (Norm-ITPO vs ITPO) (ablation) | RLOO · Medical Recommendation | diagnosis score: 68.43 -> 69.24 | Improved |
| Use token-level implicit rewards (PRIME) instead of turn-wise (ablation) | RLOO · Doc Writing | BLEU: 44.59 -> 40.95 | Hurt |
| Use token-level implicit rewards (PRIME) instead of turn-wise (ablation) | PPO value model · Math Tutoring | accuracy: 26.81 -> 22.63 | Hurt |
| Train an extra value model (PPO) instead of group-based advantages | Qwen2.5-3B-Instruct · Math Tutoring, Doc Writing, Medical Recommendation | accuracy / BLEU / diagnosis | No clear effect |
| Uniform random decomposition of outcome reward | RLOO · Math Tutoring, Doc Writing, Medical Recommendation | accuracy / BLEU / diagnosis | No clear effect |
| Use LLM-as-Judge turn weights | RLOO · Doc Writing | BLEU: 37.35 -> 42.34 | Improved |
| Apply Norm-ITPO to other policy backbones | RLOO, Qwen3-4B and Qwen2.5-7B · Doc Writing | score | Improved |
Closest papers by meaning
- InfoPO: Information-Driven Policy Optimization for User-Centric Agents (ICML 2026 · similarity 0.88)
- Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.87)
- Multi-turn Reinforcement Learning with Preference Human Feedback (NeurIPS 2024 · similarity 0.86)
- Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.86)
- PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimization (NeurIPS 2026 · similarity 0.86)
- ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
