Research map / Large language models / Reasoning and chain of thought

SD-Search: Hindsight Self-Distillation for Search-Augmented Reasoning

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SD-Search, which derives step-level credit for search queries from the policy itself via on-policy hindsight self-distillation (JSD between a hindsight-conditioned teacher and prefix-only student at query tokens) layered on GRPO.

Problem

Search-augmented reasoning agents trained with outcome-reward RL give every search query in a rollout the same trajectory-level reward, so individual queries get no step-specific credit; existing process-supervision fixes rely on external teachers or annotations.

Approach

The same policy acts as student (inference-time context) and teacher (context augmented with a hindsight block of sibling rollouts' search queries and correct/incorrect labels, with future masking); a token-level JSD at search-query positions transfers the teacher's query preferences to the student, added to GRPO without changing the advantage.

Key results

• SD-Search-Base reaches 0.428 average EM with Qwen2.5-3B, statistically matching Thinker-Instruct (0.430) and beating AutoRefine-Base (0.405) without an external teacher.
• SD-Search-Instruct reaches 0.476 average EM with Qwen2.5-7B, surpassing MR-Search-Base (0.460), AutoRefine-Base (0.455) and Thinker-Instruct (0.452).
• Ablations show future masking removal costs 3.0 points on average and random outcome labels cost 2.3 points, more than removing labels (1.4).

Surprising findings

• Randomly shuffling outcome labels hurts more (-2.3 points) than removing them (-1.4 points), indicating misleading labels are worse than none.
• Thinker's gain over AutoRefine turns negative at 7B and 14B, while SD-Search stays positive at every tested scale.

Limitations

• Relies on gold-answer-derived binary outcome labels, restricting use to tasks with scorable references.
• Hindsight contrast degenerates when the rollout group is label-homogeneous (all correct or all incorrect), narrowing gains at 14B.
• Gains over strongest process-supervision baseline are within noise at 3B; statistical testing is not formal.

Experimental setup

Datasets: NQ HotpotQA

Benchmarks: NQ TriviaQA PopQA HotpotQA 2Wiki MuSiQue Bamboogle

Metrics: exact match

Baselines: Search-R1 ReSearch AutoRefine MR-Search StepSearch Thinker Search-o1 IRCoT Naive RAG R1

Builds on: GRPO Search-R1 veRL AutoRefine E5-base-v2

Closest papers by meaning

• Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration (ICML 2026 · similarity 0.90)
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards (ICML 2026 · similarity 0.89)
• DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.88)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (NeurIPS 2026 · similarity 0.88)
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search (NeurIPS 2024 · similarity 0.88)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.88)