Research map / Large language models / Reasoning and chain of thought

Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SearchAgent-X, an inference system for LLM search agents that combines high-recall ANN retrieval, priority-aware scheduling over prefix-cached requests, and non-stall retrieval with adaptive early termination, achieving up to 3.4x throughput and 5x lower latency over vLLM-based baselines.

Problem

LLM-based search agents interleave reasoning and retrieval, which creates efficiency bottlenecks: exact retrieval is costly, coarse retrieval adds reasoning steps, and retrieval latency gets amplified by scheduling and stall effects in the serving system.

Approach

Builds a serving system around high-recall HNSW retrieval with (1) a multi-metric hierarchical priority scheduler that favors requests with longer reusable prefixes (retrieval count, context length, waiting time) to improve prefix KV-cache hits, and (2) a non-stall retrieval mechanism that terminates ANN search early when an EMA of relative candidate quality plateaus and the LLM engine is ready.

Key results

• SearchAgent-X achieves 1.3-3.4x higher throughput and 0.2-0.6x latency vs baselines in offline inference on Musique with Qwen-7B.
• In online serving at request rate 6, SearchAgent-X completes 5.8x more requests than vLLM_ENN and 1.9x more than CachevLLM_ANN.
• Generation accuracy matches exact retrieval (avg EM 0.430 for both) across six QA datasets.

Surprising findings

• Both overly high and overly low retrieval recall degrade end-to-end efficiency, with a 'less is more' effect where low-recall retrieval forces extra retrieval steps.
• Minor retrieval latency increases magnify end-to-end search agent latency by over 83x, whereas naive RAG remains stable.

Limitations

• Only dense ANN retrieval is considered; keyword or hybrid retrieval is not handled.
• Uses a fixed dense encoder; encoding optimization is not explored.
• Evaluated only on Qwen-7B and Qwen-14B with Search-R1 reasoning models and a single Wikipedia knowledge base.

Experimental setup

Datasets: Wikipedia (chunked, ~21M chunks) Musique NQ 2Wiki HotpotQA Bamboogle StrategyQA

Benchmarks: Musique NQ 2Wiki HotpotQA Bamboogle StrategyQA

Metrics: throughput latency P99 latency pending sequence ratio exact match retrieval counts prefix cache hit rate

Baselines: vLLM_ENN vLLM_ANN CachevLLM_ANN

Builds on: vLLM Search-R1 HNSW

Closest papers by meaning

• Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.91)
• AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.90)
• Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers (NeurIPS 2025 · similarity 0.90)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (NeurIPS 2026 · similarity 0.89)
• Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.89)
• Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (ICLR 2026 · similarity 0.89)