Research map / Multi-agent systems and games / LLM agents

CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CONCUR, an AIMD-style agent-level admission controller that regulates how many agents issue generation requests based on KV cache usage and hit rate, to prevent middle-phase thrashing in agentic batch inference on SGLang.

Problem

Agentic batch inference grows KV cache footprints cumulatively, causing cache hit rates to collapse (middle-phase thrashing) well before memory capacity is exhausted, with LRU eviction and recomputation degrading throughput.

Approach

Treat the agent (not the request) as the unit of admission and apply TCP-style AIMD to a congestion window of active agents, using KV cache usage as a proactive congestion signal and cache hit rate as a reactive failure signal.

Key results

• On Qwen3-32B (batch 256, TP8, 8 GPUs), CONCUR reduces end-to-end latency from 1480 s (SGLang) to 362 s, a 4.09x speedup.
• On DeepSeek-V3 (batch 40, TP16, 16 GPUs), CONCUR achieves 1.90x speedup (2043 s vs 3877 s for SGLang).
• At batch 40 for DeepSeek-V3, SGLang's KV cache hit rate falls to 35.41% while CONCUR maintains 73.36%.

Surprising findings

• Adding more agents during the middle phase reduces overall throughput even with unchanged model size and hardware.
• Request-level admission control alone performs worse than plain SGLang in some configurations (e.g., 0.72x on Qwen3-32B TP8).

Limitations

• Hyperparameters (alpha, beta, thresholds) fixed by heuristic choice; U_low requires careful calibration per sensitivity analysis.
• Evaluation limited to offline batch latency on two models and one internal workload; no public code or data stated.
• Middle-phase thrashing characterization based on a single large-scale deployment trace.

Experimental setup

Benchmarks: real-world agent workloads (internal trace-driven benchmark)

Metrics: end-to-end batch latency (s) KV cache hit rate (%) speedup

Baselines: SGLang SGLang w/ Request Control SGLang w/ HiCache fixed admission limits (30, 32, 64, 128)

Builds on: AIMD (TCP congestion control) SGLang RadixAttention prefix caching ReAct

Closest papers by meaning

• KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows (NeurIPS 2025 · similarity 0.89)
• ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management (ICML 2026 · similarity 0.88)
• LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents (ICML 2026 · similarity 0.87)
• Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents (NeurIPS 2025 · similarity 0.87)
• A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.87)
• A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems (ICML 2026 · similarity 0.87)