Research map / Multi-agent systems and games / LLM agents
Unlocking Long-Horizon Agentic Search with Large-Scale End-to-End RL
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, Yi Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a large-scale end-to-end RL recipe for training long-horizon LLM search agents without reliance on commercial-LLM trajectories or distillation.
Problem
Open-source search agents depend heavily on advanced commercial LLMs, either via SFT on their trajectories or by calling them as tools, limiting autonomy and long-horizon search capability.
Approach
Trains search agents end-to-end with large-scale reinforcement learning instead of imitating trajectories from stronger commercial models, aiming for long-horizon agentic search built from the agent's own rollouts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| end-to-end RL with 128-turn limit | QwQ-32B search agent · GAIA | LLM-as-Judge Pass@1: text says +15.0; Table 1 gives 58.7 | Improved |
| raise training turn limit from 10 to 128 (ablation) | ASearcher · GAIA | LLM-as-Judge Pass@1: 49.2 -> 58.7 | Improved |
| raise training turn limit from 10 to 128 (ablation) | ASearcher · xBench-DeepSearch | LLM-as-Judge Pass@1: 39.3 -> 51.1 | Improved |
| add stage-2 curriculum on questions needing >=5 tool calls (ablation) | ASearcher · GAIA | LLM-as-Judge Pass@1: 51.6 -> 58.7 | Improved |
| add stage-2 curriculum on questions needing >=5 tool calls (ablation) | ASearcher · xBench-DeepSearch | LLM-as-Judge Pass@1: 43.0 -> 51.1 | Improved |
| train on synthesized QA data instead of AFM data (ablation) | ASearcher · GAIA | LLM-as-Judge Pass@1: 50.9 -> 58.7 | Improved |
| train on synthesized QA data instead of AFM data (ablation) | ASearcher · xBench-DeepSearch | LLM-as-Judge Pass@1: 39.9 -> 51.1 | Improved |
| use DeepSeek-V3 as external webpage summarizer | ASearcher · GAIA | LLM-as-Judge Pass@1: 58.7 -> 60.3 | Improved |
| add test-time search aggregating K=16 rollouts | ASearcher · GAIA | LLM-as-Judge Pass@1: 60.3 -> 71.8 | Improved |
| add test-time search aggregating K=16 rollouts | ASearcher · xBench-DeepSearch | LLM-as-Judge Pass@1: 56.4 -> 75.0 | Improved |
| apply same RL recipe to a small non-reasoning model | Qwen2.5-7B-Instruct · GAIA | LLM-as-Judge Pass@1: 18.2 -> 27.7 | Improved |
| apply same RL recipe to a small non-reasoning model | Qwen2.5-7B-Instruct · xBench-DeepSearch | LLM-as-Judge Pass@1: 23.0 -> 27.8 | Improved |
Closest papers by meaning
- Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (NeurIPS 2026 · similarity 0.92)
- Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.90)
- Search Self-Play: Pushing the Frontier of Agent Capability without Supervision (ICLR 2026 · similarity 0.90)
- OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents (NeurIPS 2026 · similarity 0.89)
- ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
