Research map / Large language models / Reasoning and chain of thought

Reasoning Language Model Inference Serving Unveiled: An Empirical Study

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirical study of serving reasoning LLMs (DeepSeek-R1-Distill models) versus standard LLMs on vLLM/SGLang, characterizing memory fluctuation, stragglers, adaptive runtime, and domain preference, and testing quantization, KV quantization, prefix caching, speculative decoding, and PD disaggregation.

Problem

Serving behavior and performance of reasoning LLMs (long chain-of-thought) is unexplored; existing LLM serving optimizations may not transfer to RLLMs with long outputs and bursty workloads.

Approach

Proposes the ASU assessment framework (accuracy, service-provider throughput, user-side latency) and the ASU-Perf benchmark suite to compare RLLM and LLM serving and evaluate common inference optimizations on RLLMs, then validates findings under Gamma-distributed workloads.

Key results

• RLLM KV cache usage fluctuates between 3% and 70% while traditional LLMs typically stay below 3%.
• GPTQ-INT4 and FP8 weight quantization preserve most benchmark accuracy (within about 3%) but GPTQ drops 15-25% on AIME24; AWQ and L4 nearly double E2E time and halve throughput.
• PD disaggregation (1P1D) lowers TPS and raises running time for 7B and 14B RLLMs (e.g., 7B GSM8K TPS 713.3 to 516.9).

Surprising findings

• KV cache quantization severely degrades accuracy for 7B RLLM while 14B and 32B maintain performance.
• Prefix caching increases latency for 7B RLLMs but accelerates 14B and larger models.

Limitations

• Evaluation limited by current serving engine support; speculative decoding only n-gram due to Eagle incompatibility with RLLMs.
• Limited hardware resources; only RTX 4090, A6000, and A100 GPUs tested.
• Mostly math-focused benchmarks given RLLM specialization.

Experimental setup

Datasets: GSM8K MATH-500 AIME-2024 GPQA Diamond

Benchmarks: GSM8K MATH-500 AIME-2024 GPQA Diamond

Metrics: accuracy TPS TTFVT TTFT running time E2E latency KV cache usage output tokens

Baselines: Qwen2.5-Math-7B Qwen2.5-14B Qwen2.5-32B Llama-3.3-70B-Instruct

Builds on: vLLM SGLang BurstGPT GPTQ AWQ

Closest papers by meaning

• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.89)
• Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Efficient Multi-round LLM Inference over Disaggregated Serving (ICML 2026 · similarity 0.87)
• Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling (NeurIPS 2025 · similarity 0.87)
• Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.87)