Research map / Large language models / Reasoning and chain of thought

Cascadia: An Efficient Cascade Serving System for Large Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Cascadia is a cascade serving system that jointly optimizes GPU allocation, parallelism, and threshold-based request routing across DeepSeek model tiers via a bi-level MILP plus Chebyshev-guided routing search.

Problem

Serving LLMs with a fixed GPU pool requires balancing p95 latency and response quality; existing model-cascade serving systems ignore LLM-specific workload heterogeneity and do not co-optimize deployment with routing.

Approach

A bi-level scheduler: a MILP picks GPU allocation and per-model parallelism (DP/TP/PP) to minimize max p95 latency given routing, while a Chebyshev-guided penalty search tunes confidence thresholds to minimize latency subject to a quality requirement.

Key results

• Up to 4x lower latency deadlines (2.3x average per abstract; 2.8x average in evaluation text) and up to 5x higher throughput versus stand-alone LLMs on 4 servers of 8 H100s.
• Up to 2.5x lower latency deadlines and up to 3.3x higher throughput than CascadeServe.
• Up to 1.95x higher throughput than Sarathi-Serve (1.64x average).

Surprising findings

• Optimal cascade plans can drop the largest model entirely at lower quality requirements (e.g., trace 3 at quality 70 uses only DeepSeek-dist-7B and -70B).

Limitations

• Simulator estimates are not perfectly accurate (2-7% error on throughput), relying on profiled inputs.
• Quality estimated via a GPT-4o judge on subsampled workloads, with judging overhead included.
• Evaluated only on DeepSeek and Llama3 cascades with a small number of traces.

Experimental setup

Datasets: MT-Bench

Benchmarks: MT-Bench subsampled traces (coding, math, reasoning)

Metrics: SLO attainment (SLO scale at 95%) throughput (req/s) p95 latency

Baselines: SGLang stand-alone DeepSeek-671B SGLang stand-alone DeepSeek-dist-70B CascadeServe Sarathi-Serve

Builds on: CascadeServe FrugalGPT DistServe SGLang

Closest papers by meaning

• A Unified Approach to Routing and Cascading for LLMs (ICML 2025 · similarity 0.91)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.88)
• BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute (ICML 2025 · similarity 0.88)
• MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees (NeurIPS 2025 · similarity 0.88)
• Online Cascade Learning for Efficient Inference over Streams (ICML 2024 · similarity 0.87)
• LLM Query Scheduling with Prefix Reuse and Latency Constraints (NeurIPS 2025 · similarity 0.87)