Research map / Multi-agent systems and games / LLM agents
SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpecHop, a continuous speculation framework that keeps multiple speculative threads running over multi-hop tool-use trajectories, verifying speculator observations against target tool outputs to cut latency losslessly.
Problem
Multi-hop tool-using LLM agents incur large latency because the model must wait for each external tool observation before continuing; the goal is to accelerate trajectories without changing the trajectory the model would have taken.
Approach
Extends single-hop speculative actions to a continuous pipeline: k parallel speculative threads are launched ahead of pending target-tool calls, verified asynchronously as results arrive, committed when correct and rolled back on the first failure, with a theoretical latency model and a CLT-based starvation bound for choosing k.
Key results
- With GPT-4o as Web Search speculator on 2WikiMultihopQA, SpecHop reaches RelLat 0.60 (about 40% latency reduction) vs. optimal 0.50, with EM 69.3 vs 68.7 for standard.
- On E5 retrieval with Llama 3.1 8B speculator, SpecHop keeps EM at 72.3 on 2WikiMultihopQA while latency drops from 3.81s to 3.47s.
- Full Speculation without verification drops 2WikiMultihopQA EM from 68.7 to 38.7.
Surprising findings
- Unverified full speculation severely degrades accuracy, showing speculator-only pipelines are unreliable.
- GPT speculators are not faster than local E5 retrieval, so they were excluded for that target tool because they violate the faster-speculator assumption.
Limitations
- Verifier is a deterministic rule-based (normalization, exact match, Jaccard) check; human evaluation on only 100 decisions, with 96.2% recall.
- Theoretical analysis assumes independent, identically distributed hop latencies and a CLT approximation for starvation.
- Experiments use a single small generator (CoRAG Llama 3.1 8B) and GPT-5 only on a subset of settings.
Experimental setup
Datasets: 2WikiMultihopQA MuSiQue DeepResearch-9K KILT/Wikipedia corpus
Benchmarks: 2WikiMultihopQA MuSiQue DeepResearch-9K
Metrics: relative latency wall-clock latency (s) EM F1 speculation success probability human verifier precision/recall
Baselines: Standard sequential multi-hop execution Full Speculation
Builds on: Speculative Actions speculative decoding CoRAG
Closest papers by meaning
- Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents (ICLR 2026 · similarity 0.87)
- Attention-Level Speculation (ICML 2025 · similarity 0.86)
- Accelerating Large Language Model Reasoning via Speculative Search (ICML 2025 · similarity 0.86)
- Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation (ICML 2025 · similarity 0.85)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.85)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.