Research map / Large language models / Reasoning and chain of thought

CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CAS-Spec, a training-free cascade of self-speculative draft models built from layer-sparsity DSIA variants of the target LLM plus PLD, scheduled by a Dynamic Tree Cascade (DyTC) algorithm that routes drafts using acceptance-rate and latency estimates.

Problem

Cascade speculative decoding gives larger speedups but needs multiple separately trained draft models, while training-free on-the-fly self-speculative methods offer only modest speedups; it is also unclear how to schedule cascades of self-speculative drafts efficiently.

Approach

Construct a hierarchy of draft models from the target model itself via dynamically switchable inference acceleration (layer sparsity, Kangaroo early exit), terminate with PLD, and adaptively choose draft configurations and lengths online with DyTC, which maximizes expected speedup with a bottom-model lookahead heuristic.

Key results

• On Spec-Bench with Vicuna-7B-v1.3, CAS-Spec achieves 1.578x overall speedup, versus 1.539x for PLD and 1.064x for SWIFT; CAS-Spec with Kangaroo and PLD reaches 1.696x.
• DyTC improves average speedup by 47% over the tree-based SWIFT baseline and 73% over the VC+HC cascade (CS-Drafting-style) configuration on Vicuna-7B-v1.3.
• Speedups range from 1.10x to 2.27x across models and tasks; on Vicuna-33B-v1.3 CAS-Spec reaches 1.481x overall.

Measured effects reported in the paper

Surprising findings

• Retrieval-based PLD alone outperforms existing training-free self-speculative methods such as SWIFT and Lookahead on Spec-Bench.
• The 3-level vertical cascade was rarely selected by DyTC because the layer-sparsity draft models were too similar in quality.

Limitations

• Online estimation in DyTC adds overhead and needs a warm-up period.
• Benefit diminishes at large batch sizes because scheduling relies on per-sequence heuristics.
• Incompatible with hidden-state-dependent methods such as EAGLE3; activation quantization and sparsity DSIAs were not evaluated in the main experiments.

Experimental setup

Datasets: Spec-Bench

Benchmarks: Spec-Bench

Metrics: speedup over autoregressive decoding mean accepted tokens

Baselines: Lade PLD SWIFT Kangaroo CS-Drafting Medusa EAGLE EAGLE2

Builds on: SWIFT CS-Drafting Kangaroo SpecInfer Prompt Lookup Decoding

Closest papers by meaning

• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.91)
• LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding (ICML 2026 · similarity 0.90)
• SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.90)
• TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.90)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)