Research map / Large language models / Reasoning and chain of thought

ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ConServe, an LLM co-serving system that harvests idle GPU cycles by co-locating online and offline requests using token-level SLO-aware scheduling, layer-wise preemption, and incremental KV cache checkpointing on vLLM.

Problem

Online LLM serving is bursty and leaves GPUs underutilized, but naively co-serving latency-tolerant offline requests with online ones causes large online TTFT/TBT SLO violations and offline throughput loss due to coarse-grained scheduling, iteration-level preemption, and request-level KV cache manag

Approach

Manage resources at finer granularities: a context-aware latency model sizes offline tokens per iteration to fit TBT slack; safepoints between transformer layers allow mid-iteration preemption; and per-token incremental KV checkpointing (append-only cache) makes preemption and resumption near-zero cost.

Key results

• Across real-world workloads, ConServe achieves on average 2.2x higher offline throughput and 2.9x lower P99 online tail latency than state-of-the-art preemptive co-serving baselines (abstract).
• Llama-3.1 70B: ConServe achieves 1553 tokens/s offline throughput on mixed workload, 78% of Non-Preemptive optimum, while Sarathi-P gets 895 tokens/s.
• Ablation on Llama-3.1 8B: SLO-aware scheduler cuts P99 TBT from 375ms to 73ms; incremental KV management improves offline throughput 1.29x (8810 to 11366 tokens/s).

Surprising findings

• Preemptive co-serving with Sarathi-P can yield worse TBT than non-preemptive scheduling because preemption forces costly prefill recomputation.
• Under memory pressure, up to 69% of GPU time can be wasted on recomputation or swapping in naive preemptive co-serving.

Limitations

• Evaluated only on H100 GPUs with a single DGX node and models up to 70B.
• Relies on an offline profiler to fit the latency model; accuracy depends on profiling grid and hardware setup.
• Host memory bounds KV checkpointing; falls back to token-level recomputation when exhausted.

Experimental setup

Datasets: BurstGPT DuReader MultiNews VCSUM

Benchmarks: BurstGPT DuReader MultiNews VCSUM synthetic Gamma-process workloads

Metrics: P99 TTFT P99 TBT offline throughput (tokens/s) SLO attainment

Baselines: Online-Only (vLLM) Non-Preemptive Sarathi-Preemptive DistServe-Preemptive

Builds on: vLLM Sarathi chunked prefill PagedAttention DistServe

Closest papers by meaning

• Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.91)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.89)
• A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.88)
• ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.87)
• DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving (ICML 2024 · similarity 0.86)
• SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs (NeurIPS 2025 · similarity 0.86)