Research map / Multi-agent systems and games / LLM agents

Search Self-Play: Pushing the Frontier of Agent Capability without Supervision

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Search Self-play, where a deep search LLM agent alternates between proposing verifiable search questions (validated by RAG-based solving) and solving them, improving agent capability without human-annotated data.

Problem

RLVR for search agents needs many well-crafted queries with ground-truth answers, which is costly to produce by humans or offline synthesis; offline synthesis also cannot adapt difficulty during RL.

Approach

A single LLM plays both a question proposer (multi-turn search to build a question from a given answer) and a solver; proposed questions are validated by having the solver answer with the proposer's collected search results as RAG context (plus noisy documents), and the proposer is rewarded by the solver's failure rate, yielding a co-evolving adaptive curriculum.

Key results

• SSP on Qwen2.5-7B-Base improves average accuracy from 22.3 to 48.7 without any annotation (+26.4).
• SSP on Qwen2.5-7B-Instruct raises average from 41.5 to 49.5, outperforming Solver-Only (44.2) and Proposer-Only (41.7) variants.
• SSP on Qwen2.5-32B-Instruct reaches 58.5 average (+3.4), state-of-the-art on five of seven benchmarks per the authors.

Surprising findings

• Adding 4 randomly sampled noisy documents to RAG verification gives the best results, while 7 noisy documents degrade performance.
• Penalizing invalid proposer outputs with -0.1 causes training collapse, as the valid question rate falls toward zero.

Limitations

• Search steps capped at 10, which the authors say causes performance plateau in later training.
• Only about 50% question validation success rate by end of training, so many valid questions are filtered out.
• Evaluated only on QA benchmarks with a Wiki-2018 local retriever; the RAG verifier uses Qwen2.5-32B-Instruct as solver.

Experimental setup

Datasets: NQ HotpotQA ARPO training set Wiki-2018 corpus

Benchmarks: NQ TriviaQA PopQA HotpotQA 2WikiMultiHopQA MuSiQue Bamboogle

Metrics: pass@1 accuracy

Baselines: Qwen2.5-7B-Instruct LLaMA-3.1-8B Qwen3-8B ZeroSearch-7B Search-R1-7B R-Search-7B Qwen2.5-14B-Instruct Qwen2.5-32B-Instruct

Builds on: Search-R1 GRPO REINFORCE VeRL dynamic sampling

Closest papers by meaning

• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (NeurIPS 2026 · similarity 0.88)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.88)
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards (ICML 2026 · similarity 0.88)
• Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.87)
• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs (ICLR 2026 · similarity 0.87)
• Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.87)