Research map / Reinforcement learning / RL from feedback
TIPS: Turn-level Information-Potential Reward Shaping for Search-Augmented LLMs
Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Li Erran Li, Xiaolong Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add turn-level information-gain reward | PPO · Search QA avg (Qwen2.5-7B) | EM: 37.28 -> 41.71 | Improved |
| add turn-level information-gain reward | PPO · Search QA avg (Qwen2.5-3B) | EM: 30.15 -> 33.60 | Improved |
| add turn-level information-gain reward | PPO · Search QA avg (Qwen2.5-7B) | F1: 45.07 -> 51.24 | Improved |
| add turn-level information-gain reward | PPO · 2WikiMultiHopQA (Qwen2.5-7B) | EM: 32.94 -> 42.96 | Improved |
| add turn-level information-gain reward | PPO · NQ (Qwen2.5-3B) | EM: 43.80 -> 43.46 | No clear effect |
| use MT-GRPO turn-level rule rewards instead of GRPO | GRPO · Search QA avg (Qwen2.5-7B) | EM: 28.54 -> 30.42 | Improved |
| use MT-GRPO instead of GRPO | GRPO · Search QA avg (Qwen2.5-3B) | EM: 30.86 -> 26.35 | Hurt |
| use a different teacher (TIPS-trained Qwen3-4B) instead of frozen policy (ablation) | TIPS · Search QA avg (Qwen2.5-7B) | EM: 41.7 -> 30.0 | Hurt |
| use Llama3.1-8B teacher instead of frozen policy (ablation) | TIPS · Search QA avg (Qwen3-4B) | EM: 48.4 -> 43.0 | Hurt |
| fix the teacher (refresh N=500) rather than refresh every 200 steps (ablation) | TIPS · Search QA avg | EM | Hurt |
| use large shaping target band (0.3-1.0) instead of medium (ablation) | TIPS · Search QA avg (Qwen3-4B) | EM gain over PPO: +7.3% -> -3.4% | Hurt |
| use small shaping target band (0.001-0.05) (ablation) | TIPS · Search QA avg (Qwen2.5-7B) | EM gain over PPO: +11.9% -> 0% (crashed) | Hurt |
Closest papers by meaning
- Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.89)
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs (NeurIPS 2025 · similarity 0.89)
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.89)
- MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels (NeurIPS 2026 · similarity 0.88)
- Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
