Research map / Large language models / Reasoning and chain of thought

Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Erasable Reinforcement Learning (ERL), which detects faulty sub-answers, sub-queries or plans in search-augmented LLM rollouts, erases them, and regenerates, yielding the ESearch models that set new results on four multi-hop QA benchmarks.

Problem

Search-augmented LLMs are brittle on multi-hop QA: decomposition errors, missing retrieval, and reasoning errors propagate through the trajectory, and sparse terminal rewards do not allow localized correction.

Approach

Treats each round of search-and-reason as erasable: dense stepwise rewards (TF-IDF gold-evidence coverage with redundancy penalty, sub-answer F1 gain) trigger erasure operators that discard faulty sub-answers, search results, or the initial plan, and regenerate from the last correct state, combined with PPO.

Key results

• ESearch 3B model gains +8.48% EM and +11.56% F1 over previous SOTA as stated in the abstract (offline average gains reported as +6.06% EM and +9.94% F1).
• ESearch 7B model gains +5.38% EM and +7.22% F1 over previous SOTA per the abstract.
• ERL with PPO on Qwen2.5-7B-Base reaches 0.434 EM on HotpotQA offline versus 0.371 for PPO and 0.350 for GRPO.

Surprising findings

• GRPO learns faster early but tends to become unstable and collapse in later training stages on search-agent tasks.
• Sub-answer erasure gives the largest single-component gain, which the authors interpret as error correction during reasoning mattering more than error prevention in planning.

Limitations

• Sequential identify-erase-regenerate cycle increases computational overhead.
• Struggles when multiple heterogeneous errors occur simultaneously in one trajectory, requiring repeated iterations.
• Training and evaluation rely on Wiki-18 retrieval for most results, with online Google results on a subset of settings.

Experimental setup

Datasets: MuSiQue

Benchmarks: HotpotQA 2WikiMultiHopQA MuSiQue Bamboogle

Metrics: exact match F1

Baselines: Search-R1 ZeroSearch R-Search StepSearch SSRL Research PPO GRPO

Builds on: Search-R1 StepSearch PPO GRPO

Closest papers by meaning

• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
• Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning (ICML 2026 · similarity 0.89)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.89)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models (NeurIPS 2026 · similarity 0.88)