Research map / Large language models / Alignment and preferences
Tool-Augmented Reward Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Themis, a tool-augmented reward model that interleaves thought, tool calls, observations, and rationale before emitting a scalar reward, trained on a new 7-tool dataset (TARA).
Problem
Conventional reward models rely on static internal knowledge and struggle with arithmetic, code execution, factual lookup, and up-to-date information when scoring responses for RLHF.
Approach
Equip reward models with an agentic ReAct-style loop over external tools (calculator, search, weather, calendar, translator, code, wiki), generating thought, action, observation and rationale tokens autoregressively before a scalar reward head, trained with pairwise ranking loss plus language-modeling loss on tool trajectories.
Key results
- Themis (Vicuna-7B) improves average preference ranking accuracy on TARA from 75.63 to 93.31 in mixed-tool setting (+17.7%).
- Themis reaches 36.8 on TruthfulQA MC1 zero-shot, outperforming Gopher 280B (29.5) by 7.3 points.
- Human evaluation of PPO against Themis shows an average 32% win-rate advantage across four domains on 200 prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tool-augmented reward modeling | Vicuna-7B reward model · TARA single-tool | accuracy (Avg.): 75.04 -> 94.23 | Helped |
| Add tool-augmented reward modeling | Vicuna-7B reward model · TARA mixed-tool | accuracy (Avg.): 75.63 -> 93.31 | Helped |
| Add tool-augmented reward modeling | Vicuna-7B reward model · TruthfulQA (MC1), zero-shot | accuracy: 30.7 -> 36.8 | Helped |
| Add tool-augmented reward modeling | Vicuna-7B reward model · Retarded-bar (en) | accuracy: 68.0 -> 73.3 | Helped |
| Add tool-augmented reward modeling | Vicuna-7B reward model · WebGPT Comparison | accuracy: 66.68 -> 69.53 | Helped |
| Add tool-augmented reward modeling | Vicuna-7B reward model · HH-RLHF | accuracy: 72.34 -> 77.59 | Helped |
| Optimize the policy against Themis rather than a vanilla RM | Vicuna-7B PPO policy · RLHF PPO | PPL: 8.10 -> 7.88 | Helped |
| Add observation loss (ablation) | Themis Vicuna-7B · TARA single-tool | accuracy (Avg.): 90.23 -> 94.23 | Helped |
| Add observation loss (ablation) | Themis Vicuna-7B · TARA mixed-tool | accuracy (Avg.): 90.90 -> 93.31 | Helped |
| Add rationale loss (ablation) | Themis Vicuna-7B · TARA mixed-tool | accuracy (Avg.): 89.31 -> 93.31 | Helped |
| Add observation loss (ablation) | Themis Vicuna-7B · HH-RLHF* zero-shot | accuracy: 55.83 -> 55.00 | Hurt |
| Add observation loss (ablation) | Themis Vicuna-7B · HH-RLHF* fine-tuning | accuracy: 71.67 -> 70.00 | Hurt |
Surprising findings
- Vanilla RM (Vicuna-7B) reaches best performance after a single epoch while Themis needs more epochs to learn tool invocation.
Limitations
- Only seven tools and single-turn data; limited tool scope.
- Experiments mainly on Vicuna-7B; scaling beyond 100B untested.
- Tool invocation adds time overhead and dependence on API reliability; data generation relies on GPT-4 and heuristics.
Experimental setup
Datasets: TARA HH-RLHF WebGPT Comparisons GSM8K HumanEval MBPP MLQA Natural Questions TruthfulQA
Benchmarks: TARA HH-RLHF* TruthfulQA Retarded-bar (en) WebGPT Comparisons HH-RLHF
Metrics: accuracy perplexity win rate MC1 accuracy
Baselines: RM (Bert-Large) RM (Vicuna-7B) Gopher 280B GPT-3 OPT 175B Galactica RAFT (LLaMA-7B) Deberta-v3-large-v2
Builds on: ReAct Chain-of-Thought RLHF LoRA PPO
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.91)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.91)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.91)
- Rectifying Shortcut Behaviors in Preference-based Reward Learning (NeurIPS 2025 · similarity 0.91)
- Scaling Reward Modeling without Human Supervision (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.