Research map / Large language models / Agents, code and math
SGLang: Efficient Execution of Structured Language Model Programs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SGLang, a Python-embedded language and runtime for structured LM programs that speeds execution via RadixAttention KV cache reuse, compressed FSM constrained decoding, and API speculative execution.
Problem
LM programs with multiple dependent generation calls, control flow, and structured outputs are hard to program and execute inefficiently, since existing inference engines discard KV cache across calls and decode constrained outputs one token at a time.
Approach
Treats the KV cache as an LRU cache organized in a radix tree so that prefixes are automatically shared across calls and forks, schedules requests by longest shared prefix, compresses singular-transition FSM edges to decode multiple tokens per step, and co-designs a frontend language with the runtime.
Key results
- SGLang improves throughput by up to 6.4x and reduces latency by up to 3.7x versus Guidance, vLLM, and LMQL on open-weight models.
- On LLaVA multimodal models, SGLang gives 1.15 image/s vs 0.18 for the authors' original implementation on LLaVA-v1.5-7B.
- Compressed FSM increases JSON decoding throughput by 1.6x; RadixAttention overhead is under 0.3% when there are no cache hits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RadixAttention KV-cache reuse | SGLang · llava-bench-in-the-wild | throughput: 0.18 image/s -> 1.15 image/s | Helped |
| add RadixAttention KV-cache reuse | SGLang · ActivityNet | throughput: 0.02 frame/s -> 0.10 frame/s | Helped |
| use SGLang runtime optimizations | LM program workloads · Various LM program workloads | throughput: up to 6.4× | Helped |
| use SGLang runtime optimizations | LM program workloads · Various LM program workloads | latency: reduces latency by up to 3.7× | Helped |
| add a compressed finite state machine for constrained decoding | JSON decoding · JSON decoding benchmark | throughput: +1.6× throughput | Helped |
| redo FSM preprocessing for each request instead of reusing the preprocessed state machine for a batch (ablation) | Compressed finite state machine · JSON decoding benchmark | throughput: 2.4× lower | Hurt |
| remove KV-cache reuse (ablation) | RadixAttention · Representative RadixAttention benchmarks | performance | Hurt |
| replace the radix tree with a simple table-based cache (ablation) | RadixAttention · Representative RadixAttention benchmarks | performance | Hurt |
| replace cache-aware scheduling with FCFS scheduling (ablation) | RadixAttention · Representative RadixAttention benchmarks | performance | Hurt |
| replace cache-aware scheduling with random scheduling (ablation) | RadixAttention · Representative RadixAttention benchmarks | performance | Hurt |
| remove frontend interpreter parallelism (ablation) | SGLang · Representative RadixAttention benchmarks | runtime performance | Hurt |
| remove frontend fork hints (ablation) | SGLang · Representative RadixAttention benchmarks | runtime performance | Hurt |
Surprising findings
- Cache-aware scheduling approaches 96% of the optimal cache hit rate on average.
- Greedy cache-aware scheduling can lead to starvation.
Limitations
- Greedy cache-aware scheduling can cause starvation.
- Compressed FSM can distort token probabilities when choices are multi-token and not aligned with tokenization.
- Compiler mode only supports programs without data-dependent control flow.
Experimental setup
Datasets: MMLU HellaSwag GSM-8K llava-bench-in-the-wild ActivityNet ShareGPT
Benchmarks: MMLU HellaSwag ReAct agent traces generative agents traces GSM-8K tree-of-thought skeleton-of-thought tip generation JSON decoding multi-turn chat DSPy RAG llava-bench-in-the-wild ActivityNet
Metrics: throughput (programs per second) latency cache hit rate images/s and frames/s
Baselines: Guidance vLLM LMQL
Builds on: vLLM PagedAttention continuous batching Guidance LMQL
Closest papers by meaning
- Recursive Language Models (NeurIPS 2026 · similarity 0.86)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.86)
- REASONING COMPILER: LLM-Guided Optimizations for Efficient Model Serving (NeurIPS 2025 · similarity 0.86)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.85)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.85)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.