Research map / Multi-agent systems and games / LLM agents

IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a Qwen2.5-7B proactive clarification agent with a two-stage RL pipeline (offline hindsight RL on C-DAG-derived dialogues, then online rollouts with an intent-aware user simulator) to elicit latent user intent before open-ended deep research.

Problem

Deep research agents run long, costly executions on ambiguous queries, and their built-in clarification modules fail to elicit sufficient latent user intent; training data for open-ended proactive clarification is scarce.

Approach

Expands a few seed queries into a clarification DAG (C-DAG) built from shallow intents (removed constraints) and rubric-derived deep intents, traverses it to generate offline expert dialogues, then trains with turn-level GRPO using embedding-based content reward and repetition/relevance penalties, followed by online RL with a hybrid rule-based and LLM-judge user simulator.

Key results

• On DeepResearch Bench with OpenAI Deep Research, IntentRL clarify raises overall RACE score from 36.62 (no clarify) to 44.86, above OpenAI's built-in clarify (40.26).
• In online simulated clarification on DeepResearch Bench, IntentRL reaches 36.44% intent precision and 27.49% intent recall, versus 18.00% and 12.68% for CollabLLM.
• Ablation: removing Stage I training drops Quality Score to 1.65 and overall report score to 39.81, versus 43.65 for full IntentRL.

Surprising findings

• Built-in clarification modules of closed-source DR agents (Qwen clarify, Tell Me More) sometimes underperform no-clarify baselines on some metrics.
• On Rigorous Bench, most proactive baselines give only marginal gains, attributed to tasks requiring up-to-date knowledge.

Limitations

• Training uses only 50 seed queries from DeepResearch Bench, and evaluation relies on simulated users generated by LLMs.
• Evaluation depends on LLM-based judges (RACE, semantic drift, P-Score) and proprietary models like gemini-3-pro-preview for simulation.
• Gains on Rigorous Bench are marginal for most baselines and the method's benefit on time-sensitive tasks is unclear.

Experimental setup

Datasets: DeepResearch Bench Rigorous Bench PDR-Bench

Benchmarks: DeepResearch Bench Rigorous Bench PDR-Bench

Metrics: Quality Score Intent Precision Intent Recall RACE (Comp., Insight, Instr., Read., Overall) Semantic Quality 1-SDrift P-Score Q-Score

Baselines: Qwen Deep Research clarify OpenAI Deep Research clarify Learn-to-Ask Tell Me More CollabLLM SFT

Builds on: GRPO CollabLLM Trinity-RFT

Closest papers by meaning

• Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization (ICML 2026 · similarity 0.88)
• Simulation-Free Hierarchical Latent Policy Planning for Proactive Dialogues (AAAI 2025 · similarity 0.87)
• Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.87)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents (ICLR 2024 · similarity 0.86)
• Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward (NeurIPS 2025 · similarity 0.86)
• Decoupling Conversational Dynamics in Full-Duplex Spoken Models through Reinforcement Learning (NeurIPS 2026 · similarity 0.86)