Research map / Large language models / Agents, code and math
RF-Agent: Automated Reward Function Design via Language Agent Tree Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RF-Agent, which treats an LLM as a language agent and uses Monte Carlo Tree Search with LLM-based expansion actions and self-verify scores to design dense reward functions for low-level control tasks.
Problem
LLM-based reward function design for low-level control tasks (IsaacGym, Bi-DexHands) is limited by poor use of historical feedback and inefficient greedy or evolutionary search.
Approach
Frame reward design as sequential decision-making over a search tree: MCTS with an improved UCT (normalized Q, exploration term, LLM self-verify score), five LLM action types (two mutations, crossover over elites, path reasoning over ancestors, different-thought) that use whole-tree history, and a thought-alignment step after compilation.
Key results
- On IsaacGym with GPT-4o, RF-Agent reaches an average normalized score of 2.68 versus 2.00 for Eureka and 2.03 for Revolve.
- With GPT-4o-mini on IsaacGym, RF-Agent achieves 1.70 average normalized score versus 0.63 (Eureka) and 0.67 (Revolve).
- On AllegroHand with GPT-4o-mini, RF-Agent scores 22.52 versus 13.87 (Eureka) and 18.40 (Revolve).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RF-Agent instead of the strongest listed LLM-based baselines | LLM-based reward function design · IsaacGym, GPT-4o-mini-0718 | Avg norm score: 0.67 -> 1.70 | Helped |
| Use RF-Agent instead of the strongest listed LLM-based baselines | LLM-based reward function design · IsaacGym, GPT-4o-0806 | Avg norm score: 2.03 -> 2.68 | Helped |
| Replace improved UCT selection with DFS (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace improved UCT selection with BFS (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace improved UCT selection with Greedy search (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace mutation action a_m1 with the basic parent-state action (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace mutation action a_m2 with the basic parent-state action (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace crossover action with the basic parent-state action (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace path reasoning with the basic parent-state action (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Replace different-thought action with the basic parent-state action (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Remove the designed action types and use only the basic action (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
| Remove self-verify (ablation) | RF-Agent · Ant, FrankaCabinet, and AllegroHand | performance | Hurt |
Surprising findings
- Even the lightweight GPT-4o-mini backbone with RF-Agent outperforms human expert reward functions on nearly all IsaacGym tasks, while Eureka and Revolve do not, particularly on locomotion.
Limitations
- Computationally expensive and time-consuming due to many LLM calls and repeated RL training runs.
- Does not reduce the number of RL training cycles needed.
Experimental setup
Datasets: IsaacGym tasks (Ant, Anymal, Humanoid, Quadcopter, AllegroHand, FrankaCabinet, ShadowHand) Bi-DexHands tasks (10 tasks)
Benchmarks: IsaacGym Bi-DexHands
Metrics: task score / success rate human-normalized average score
Baselines: Eureka Revolve Human reward Sparse reward
Builds on: Eureka Revolve LATS PPO
Closest papers by meaning
- R*: Efficient Reward Design via Reward Structure Evolution and Parameter Alignment Optimization with Large Language Models (ICML 2025 · similarity 0.89)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.89)
- Progress Reward Model for Reinforcement Learning via Large Language Models (NeurIPS 2025 · similarity 0.88)
- Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.88)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.88)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.