Research map / Reinforcement learning / RL from feedback
ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArCHer, a hierarchical actor-critic framework for multi-turn LLM agent RL that trains an utterance-level TD critic and a token-level policy-gradient actor, achieving far better sample efficiency than token-level PPO on language agent tasks.
Problem
Training LLM agents for multi-turn tasks with delayed long-horizon rewards is hard: on-policy PPO is sample-inefficient, token-level off-policy methods suffer long horizons, and utterance-level methods face huge action spaces for maximization.
Approach
A two-level hierarchical MDP: an off-policy TD-trained utterance-level Q/V critic at the high level, and a token-level on-policy policy gradient actor at the low level that uses the high-level advantage as its terminal reward, so the critic avoids token-level Bellman backups and maximization over huge utterance spaces.
Key results
- ArCHer reaches a Twenty Questions return above -17 with fewer than 1000 samples versus over 100k for PPO, reported as at least a 100x sample-efficiency gain.
- On WebShop, online ArCHer with a GPT-2 base outperforms expert-prompt and ReAct prompting on GPT-3.5 (gpt-3.5-turbo-instruct).
- In the offline setting on Twenty Questions, ArCHer (IQL + AWR) achieves -14.1 return versus -15.4 for filtered BC and -16.8 for BC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IQL + AWR instead of IQL + REINFORCE (ablation) | ArCHer offline · Twenty Questions | return: -20 -> -14.1 | Helped |
| Add a behavior-cloning loss to IQL + REINFORCE (ablation) | ArCHer offline · Twenty Questions | return: -20 -> -15.3 | Helped |
| Use AWR instead of REINFORCE for the actor (ablation) | ArCHer with IQL critic · Twenty Questions | return: -15.3 -> -14.1 | Helped |
| Use IQL instead of SARSA for the critic (ablation) | ArCHer with AWR actor · Twenty Questions | return: -14.5 -> -14.1 | Helped |
| Use IQL + AWR rather than BC | ArCHer offline · Twenty Questions | return: -16.8 -> -14.1 | Helped |
| Use IQL + AWR rather than filtered behavior cloning | ArCHer offline · Twenty Questions | return: -15.4 -> -14.1 | Helped |
| Reuse off-policy data rather than rely on PPO's on-policy rollouts | ArCHer versus PPO · Twenty Questions | samples to average return just higher than -17: more than 100k samples -> fewer than 1000 samples | Helped |
| Use a replay buffer containing only the most recent 48 rollouts (ablation) | ArCHer · Guess My City | learning stability | Hurt |
| Increase replay-buffer size beyond the point needed for stable learning (ablation) | ArCHer · Guess My City | performance | No clear effect |
| Replace the RoBERTa critic with a GPT-2 critic (ablation) | ArCHer · Online tasks | performance | No clear effect |
| Use a Mistral 7B actor instead of a GPT-2 actor (ablation) | ArCHer · Twenty Questions Subset | learning speed | Helped |
| Add a token-level value baseline to the policy gradient (ablation) | ArCHer · Guess My City | performance | Helped |
Surprising findings
- Token-level DQN could not attain non-zero task performance and was omitted.
- ReAct prompting underperformed significantly with gpt-3.5-turbo-instruct compared with the original text-davinci-002 model.
Limitations
- Most experiments use a small GPT-2 (100M) actor due to compute; Mistral-7B evaluated only on one task.
- Evaluated only on tasks with computational rewards; requires thousands of interactions, impractical with live humans.
- Oracle models (flan-t5-small) are imperfect, enabling reward hacking, e.g., in Guess My City.
Experimental setup
Datasets: Detective Game Twenty Questions Guess My City WebShop
Benchmarks: Detective Game Twenty Questions Twenty Questions Subset Guess My City WebShop
Metrics: task return sample efficiency (samples to reach return)
Baselines: PPO Filtered BC CHAI ILQL ReAct BC
Builds on: PPO REINFORCE implicit Q-learning advantage-weighted regression CHAI
Closest papers by meaning
- LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models (ICML 2025 · similarity 0.91)
- Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF (ICLR 2025 · similarity 0.91)
- Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL (NeurIPS 2025 · similarity 0.90)
- Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning (ICLR 2025 · similarity 0.90)
- Multi-turn Reinforcement Learning with Preference Human Feedback (NeurIPS 2024 · similarity 0.89)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.