Research map / Large language models / Agents, code and math
ThunderAgent: A Fast, Simple, and Program-Aware Agentic Inference System
Hao Kang, Ziyang li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use program-aware scheduler with state-aware pausing instead of request-level eviction (vLLM) | ThunderAgent · serving, 96 parallel programs | throughput: 1.48-3.58x over vLLM | Improved |
| use time-decay function f(t) rather than static TTL pinning for acting programs (ablation) | ThunderAgent · stochastic tool execution (ToolOrchestra on HLE) | throughput | Improved |
| replace session-based node pinning with global program-aware waiting queue (ablation) | ThunderAgent · distributed rollout, 2 to 8 nodes | speedup over SGLang cache-aware routing: 1.79x at 2 nodes -> 2.39x at 8 nodes | Improved |
| add hook-based garbage collection and asynchronous environment preparation (ablation) | ThunderAgent · OpenHands rollout | disk memory and latency: 4.2x disk savings, about 10% latency improvement | Improved |
| increase detection period Delta t (ablation) | ThunderAgent · mini-SWEAgent, GLM4.6, single H100 | throughput | Hurt |
| increase base x of decay f(t)=x^-t (more aggressive eviction) (ablation) | ThunderAgent · mini-SWEAgent, GLM4.6, single H100 | throughput | Hurt |
| vary Delta t and decay base across settings (ablation) | ThunderAgent · mini-SWEAgent, GLM4.6 | throughput | No clear effect |
| add KV cache offloading (vLLM + LMCache) as a remedy for memory pressure | vLLM · GLM-4.6 with mini-SWEAgent | throughput | Hurt |
| apply prefill-decode disaggregation to agentic workloads | vLLM · agentic serving | thrashing / throughput | Hurt |
| add ThunderAgent on top of SGLang HiCache offloading | SGLang HiCache · MiniMax M2.5 on R2E-Gym, 8xH100, batch 64-192 | throughput and latency: P50 latency flat at ~10 s vs ~30 s at batch 160 and ~70 s at batch 192 | Improved |
| run ThunderAgent on A100 at low concurrency (24) | ThunderAgent · 8xA100, mini-SWEAgent and OpenHands | throughput | No clear effect |
Closest papers by meaning
- Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows (NeurIPS 2026 · similarity 0.89)
- KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows (NeurIPS 2025 · similarity 0.89)
- KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems (NeurIPS 2025 · similarity 0.88)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.88)
- A Two-Tier Perspective on Inference-Time Parallelism in Multi-Agent LLM Systems (ICML 2026 · similarity 0.88)
- CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
