Research map / Multi-agent systems and games / LLM agents

ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReSeek, a search-agent RL framework with a JUDGE action and dense reranker-based self-correction reward, plus the contamination-resistant FictionalHot benchmark.

Problem

RL-trained search agents rely on sparse or rule-based outcome rewards, so early erroneous search paths cascade into wrong answers without recovery; existing QA benchmarks also risk data contamination.

Approach

Adds an internal JUDGE action after each retrieval where the agent labels evidence as useful or not, and trains it with a dense reward that compares its judgment to an ideal label derived from bge-reranker similarity between the observation and the ground-truth answer, with asymmetric penalties.

Key results

• ReSeek with Qwen2.5-7B-Instruct reaches 0.377 average EM vs 0.346 for ZeroSearch and 0.342 for Search-R1.
• ReSeek with Qwen2.5-3B-Instruct reaches 0.312 average EM vs 0.281 for ZeroSearch.
• On FictionalHot, ReSeek scores 0.061 (7B) and 0.059 (3B), with direct inference near zero (0.001).

Surprising findings

• Reranker-only intervention without RL training already improves Search-R1 by 1.2 average EM points.
• Performance on FictionalHot is near zero for all methods, and the paper reports a 0.003 score on the Google-search variant, indicating low absolute scores on this benchmark.

Limitations

• The judge signal is derived from similarity to the ground-truth answer, which may be weaker for deep-search tasks with indirect evidence.
• Gains over Search-R1 on single-hop and some multi-hop datasets are small; ZeroSearch is competitive on single-hop tasks.
• FictionalHot consistency audit found a small fraction of inconsistent or ambiguous instances.

Experimental setup

Datasets: NQ HotpotQA PopQA TriviaQA 2WikiMultiHopQA Musique Bamboogle FictionalHot

Benchmarks: NQ TriviaQA PopQA HotpotQA 2Wiki Musique Bamboogle FictionalHot

Metrics: exact match

Baselines: Search-R1 ZeroSearch Search-o1 R1 RAG HiPRAG

Builds on: Search-R1 GRPO FlashRAG Search-o1

Closest papers by meaning

• Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration (ICML 2026 · similarity 0.92)
• SD-Search: Hindsight Self-Distillation for Search-Augmented Reasoning (NeurIPS 2026 · similarity 0.89)
• Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning (NeurIPS 2025 · similarity 0.89)
• MAGE: Multi-Agent Self-Evolution with Co-Evolutionary Knowledge Graphs (NeurIPS 2026 · similarity 0.88)
• Search Self-Play: Pushing the Frontier of Agent Capability without Supervision (ICLR 2026 · similarity 0.88)
• Language Models can Self-Improve at State-Value Estimation for Better Search (NeurIPS 2025 · similarity 0.88)