Research map / Large language models / Reasoning and chain of thought

Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Search-R2, an Actor-Refiner framework where a shared-weight Meta-Refiner (discriminator + trimmer) performs cut-and-regenerate repair of search-integrated reasoning trajectories, trained end-to-end with GRPO on a hybrid outcome+process reward.

Problem

Training search-integrated reasoning agents with RL suffers from multi-scale credit assignment: sparse trajectory-level outcome rewards cannot distinguish good reasoning from lucky guesses or locate the step where search or reasoning first went wrong.

Approach

An Actor generates trajectories; a Meta-Refiner (sharing weights with the Actor, steered by prompts) accepts or rejects each trajectory via a discriminator and, if rejected, uses a trimmer to pick a cut-point and regenerates the suffix. Training uses GRPO over the whole interaction trace with a reward gated multiplicatively by a retrieved-evidence utility (process) signal. A mixture-policy analysi

Key results

• Search-R2 (Qwen2.5-32B-Instruct) reaches 50.8 average EM across seven datasets vs 45.6 for Search-R1 (Qwen2.5-32B).
• Search-R2 on Qwen2.5-7B reaches 40.4 average EM, above Search-R1 on Qwen3-8B (40.0) per the table.
• Search-R2 adds only 2.43% to 8.66% training time per step relative to Search-R1 across the three backbones.

Surprising findings

• Search-R2 with n=5 and one revision outperforms Search-R1 with doubled rollouts (n=10), suggesting targeted refinement beats brute-force sampling.
• Max revision 4 nearly matches the fully optimized single-revision Search-R2 (50.9 vs 50.8), implying joint optimization distills extra revisions into the policy.

Limitations

• Process reward relies on an external judge whose identity and cost are not clearly specified in the main text.
• Theoretical gain conditions (positive covariances) are stated as assumptions and not empirically verified beyond aggregate results.
• Only one revision by default; gains saturate with more revisions.

Experimental setup

Datasets: NQ HotpotQA

Benchmarks: NQ TriviaQA PopQA HotpotQA 2WikiMultiHopQA Musique Bamboogle

Metrics: exact match win/fail counts on trajectory rubrics training time per step

Baselines: Direct Inference CoT IRCoT Search-o1 RAG SFT R1-base R1-instruct Rejection Sampling Search-R1

Builds on: Search-R1 GRPO rejection sampling IRCoT Search-o1

Closest papers by meaning

• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards (ICML 2026 · similarity 0.92)
• SD-Search: Hindsight Self-Distillation for Search-Augmented Reasoning (NeurIPS 2026 · similarity 0.90)
• Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (ICML 2024 · similarity 0.89)
• A$^2$FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning (ICLR 2026 · similarity 0.89)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.89)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.89)