Research map / Large language models / Reasoning and chain of thought
DeepDiver: Adaptive Web-Search Intensity Scaling via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WebPuzzle, a 24k-train/275-test open-web information-seeking benchmark, and DeepDiver, a cold-start SFT plus GRPO pipeline that trains 7B models to scale search intensity on live web search.
Problem
LLMs struggle with iterative evidence gathering in open-web question answering, and existing prompting, SFT and RL methods are mostly trained and evaluated on clean, wiki-based sources like HotpotQA.
Approach
An RL framework (DeepDiver) on a noisy open-web benchmark (WebPuzzle) with difficulty-based data mixing, an extra search-call bonus that fades during training, and a loose-to-strict LLM grader schedule, yielding emergent search intensity scaling (more search rounds on harder problems).
Key results
- DeepDiver-Qwen2.5-7B reaches 37.6 accuracy on WebPuzzle vs 29.8 for R1-Distill and 32.7 for DeepSeek-R1 (no search), and 37.6/38.1 for Qwen/Pangu DeepDiver variants on WebPuzzle.
- Search rounds for DeepDiver-Pangu-7B rise from 1.84 (cold-start SFT) to 2.89 with accuracy from 30.3 to 38.1 on WebPuzzle.
- On ProxyQA, DeepDiver reaches 32.72% vs 23.25% for R1-Distill, with longer responses (1971.58 vs 590.31 tokens).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with DeepDiver RL after cold-start SFT | Qwen2.5-7B-Instruct · WebPuzzle | accuracy: 27.9 -> 37.6 | Helped |
| Train with DeepDiver RL after R1 distillation | Qwen2.5-7B-Instruct · WebPuzzle | accuracy: 29.8 -> 37.6 | Helped |
| Train with DeepDiver RL after cold-start SFT | Pangu-7B-Reasoner · WebPuzzle | accuracy: 30.3 -> 38.1 | Helped |
| Train with DeepDiver RL after R1 distillation | Pangu-7B-Reasoner · BamBoogle-zh-71 | accuracy: 53.5 -> 69.5 | Helped |
| Continue training with strict rather than loose rewards after step 80 (ablation) | DeepDiver-Qwen7B · WebPuzzle | accuracy: 29.1 to 37.6 | Helped |
| Continue training with loose rewards rather than switching to strict rewards (ablation) | DeepDiver-Qwen7B · WebPuzzle | accuracy | No clear effect |
| Continue training with loose rewards (ablation) | DeepDiver-Qwen7B · FRAMES-zh-230 | accuracy: 7-point performance drop | Hurt |
| Train with DeepDiver RL rather than R1 distillation | Qwen2.5-7B-Instruct · ProxyQA | accuracy: 23.25% -> 32.72%; 9.47 percentage points | Helped |
| Add iterative RAG (ablation) | DeepSeek-R1 · Open Riddle fraction of WebPuzzle | accuracy: 32.9 to 31.0 | Hurt |
| Use DeepDiver rather than DeepSeek-R1 with iterative RAG | WebPuzzle hard and outlier fractions · Hard&Outliers-179 | accuracy: 24.2 -> 27.9 | Helped |
Surprising findings
- The extra search-call bonus fires on only 4.5% of trajectories early and fades to 0.1% by steps 70-80, while tool-use rounds grow later, suggesting search intensity is not solely reward-driven.
- DeepSeek-R1 with iterative RAG drops from 32.9 to 31.0 on the Wiki Riddle subset, attributed to knowledge conflicts with open-web evidence.
Limitations
- Results limited to 7B models and 20k sequence length; scaling behavior untested.
- WebPuzzle curation relies on DeepSeek-R1, introducing potential bias; future LLMs may internalize its knowledge.
- Over-searching: DeepDiver makes many more search calls even on simple tasks.
Experimental setup
Datasets: WebPuzzle C-SimpleQA-500 FRAMES-zh-230 BamBoogle-zh-71 HotpotQA Bamboogle FRAMES ProxyQA
Benchmarks: WebPuzzle C-SimpleQA-500 FRAMES-zh-230 BamBoogle-zh-71 WebPuzzle-en Bamboogle FRAMES HotpotQA ProxyQA
Metrics: accuracy average search call rounds average search queries response length
Baselines: DeepSeek-R1 QwQ-32B GPT-4o R1-Distill R1-Searcher DeepResearcher Prompted with Iterative RAG
Builds on: GRPO PPO iterative RAG DeepSeek-R1 distillation R1-Searcher DeepResearcher
Closest papers by meaning
- Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.89)
- Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging (NeurIPS 2025 · similarity 0.88)
- Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.87)
- Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels (ICLR 2026 · similarity 0.87)
- HierSearch: A Hierarchical Enterprise Deep Search Framework Integrating Local and Web Searches (AAAI 2026 · similarity 0.87)
- Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.