Research map / Reinforcement learning / RL from feedback
ARIA: Training Language Agents with Intention-driven Reward Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARIA, which clusters free-form language actions into intention groups via hierarchical clustering and aggregates rewards within clusters to train LLM agents with offline REINFORCE, evaluated on Twenty Questions, Guess My City, Bargaining, and Negotiation.
Problem
Open-ended language action spaces are combinatorially huge, so sampled actions receive extremely sparse, high-variance rewards that hinder RL training of language agents.
Approach
Project natural-language actions and observations into a low-dimensional intention space via hierarchical clustering of sentence embeddings, average returns across actions sharing a cluster to densify rewards, and choose the granularity with a reward-oriented SplitScore rather than geometric cluster metrics.
Key results
- On adversarial games, ARIA (Iter 3) reaches 58.01 average win rate on Bargaining and 48.80 on Negotiation, versus 48.36 for ArCHer and 50.13 for Step-wise DPO on Bargaining (best baseline).
- On single-agent games, ARIA (Iter 3) achieves average reward 35.25 (34.50 Twenty Questions, 36.00 Guess My City) versus 22.25 for the best baseline (Traj-wise DPO).
- Reward aggregation reduces reward variance across all four tasks, and ARIA is reported to outperform baselines by an average of 9.95% overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use intention-space reward aggregation with offline REINFORCE | Llama-3-8B-Instruct language agent · Bargaining and Negotiation | win rate: surpassing offline and online baselines by 9.67% and 9.83%, respectively | Helped |
| Use intention-space reward aggregation with offline REINFORCE | Llama-3-8B-Instruct language agent · Twenty Questions and Guess My City | average reward: 9.82% | Helped |
| Continue ARIA training from Iter 1 to Iter 2 and Iter 3 | ARIA · Bargaining, Negotiation, Twenty Questions, and Guess My City | performance: additional gains of 3.27% and 1.85% after two and three iterations, respectively | Helped |
| Apply semantic-level reward aggregation | Vanilla REINFORCE · Adversarial games | training loss | Helped |
| Apply semantic-level reward aggregation | ARIA reward-shaping variants · Bargaining and Negotiation | win rate: 17.91% and 13.80% | Helped |
| Use ε=0.1 instead of ε=0.01 for SplitScore stopping (ablation) | ARIA · Bargaining | win rate: 53.15 -> 43.86 | Hurt |
| Use ε=0.1 instead of ε=0.01 for SplitScore stopping (ablation) | ARIA · Negotiation | win rate: 45.79 -> 38.02 | Hurt |
| Use ε=0.001 instead of ε=0.01 for SplitScore stopping (ablation) | ARIA · Bargaining | win rate: 53.15 -> 46.63 | Hurt |
| Use ε=0.001 instead of ε=0.01 for SplitScore stopping (ablation) | ARIA · Negotiation | win rate: 45.79 -> 35.77 | Hurt |
| Apply ARIA reward aggregation | Qwen2.5-7B-Instruct · Bargaining | win rate: 37.92 -> 65.96 | Helped |
| Apply ARIA reward aggregation | Qwen2.5-7B-Instruct · Negotiation | win rate: 35.50 -> 47.06 | Helped |
| Apply ARIA reward aggregation | Qwen2.5-1.5B-Instruct · Bargaining | win rate: 0.02 -> 0.01 | Hurt |
Surprising findings
- Qwen2.5-1.5B-Instruct achieves near-zero win rate on Bargaining (0.02 vanilla, 0.01 ARIA), indicating the method does not help very small models on that task.
Limitations
- Effectiveness depends on the quality of sentence embeddings for capturing fine-grained behavioral differences.
- Assumes intentions are discrete and well separated, which may not hold for tasks with overlapping goals.
- Evaluated only on four text games with Llama-3-8B-based policies and LoRA fine-tuning.
Experimental setup
Datasets: Twenty Questions Guess My City Bargaining Negotiation
Benchmarks: Twenty Questions Guess My City Bargaining Negotiation
Metrics: average reward average win rate policy gradient variance
Baselines: Behavior Cloning Trajectory-wise DPO Step-wise DPO SPAG ArCHer StarPO Vanilla Model
Builds on: REINFORCE ArCHer DPO SPAG
Closest papers by meaning
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.88)
- Coupled Variational Reinforcement Learning for Language Model General Reasoning (ICML 2026 · similarity 0.87)
- Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space (NeurIPS 2026 · similarity 0.87)
- Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.87)
- Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.87)
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.