Research map / Large language models / Reasoning and chain of thought
Cost-Aware Dynamic Tree Construction for Efficient Large Language Model Inference
Yinrong Hong, Zhiquan Tan, Kai Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a cost-aware dynamic tree construction for speculative decoding that accounts for system-level costs, improving on EAGLE-2 and EAGLE-3 tree structures.
Problem
LLM autoregressive inference is latency-bound; existing dynamic-tree speculative decoding methods (EAGLE-2, EAGLE-3) neglect system-level costs of verifying larger token trees.
Approach
Builds the draft token tree dynamically while explicitly accounting for the system cost (verification/computation cost) of tree size, rather than optimizing only expected acceptance length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic reranking (cost-aware choice of nodes to verify) (ablation) | EAGLE-3 · ablation, batch 16 | speedup: 1.35x -> 2.17x | Improved |
| add dynamic reranking (ablation) | EAGLE-3 · ablation, batch 1 | speedup: 3.99x -> 3.99x | No clear effect |
| add depth pruning on top of DR (ablation) | CAST · ablation, batch 8 | speedup: 2.77x -> 2.84x | Improved |
| add breadth pruning on top of DR and DP (ablation) | CAST · ablation, batch 16 | speedup: 2.27x -> 2.35x | Improved |
| use cost-aware tree instead of EAGLE-3 tree | EAGLE-3 · Vicuna-13B HumanEval batch 1 T=0 | speedup: 4.73x -> 5.18x | Improved |
| use cost-aware tree instead of EAGLE-3 under batching | EAGLE-3 · Vicuna-13B HumanEval batch 8 T=0 | speedup: 1.91x -> 3.12x | Improved |
| use cost-aware tree vs EAGLE-3 on GSM8K Vicuna-13B | EAGLE-3 · Vicuna-13B GSM8K batch 1 T=0 | speedup: 4.00x -> 3.98x | Hurt |
| increase the verified-token limit m | EAGLE-3 · accept length vs speedup | speedup | Hurt |
| use CAST on other GPUs (H20, RTX 4090) | EAGLE-3 · batch 1 | speedup: 6.46x -> 6.66x (H20, L33 70B HumanEval) | Improved |
| increase batch size to 32 | EAGLE · HumanEval | speedup: close to 1x | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- EAGLE-3 — EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025) · better on 13 of 13 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Metrics: inference latency
Baselines: EAGLE-2 EAGLE-3
Builds on: EAGLE-2 EAGLE-3
Closest papers by meaning
- Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.91)
- TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding (NeurIPS 2026 · similarity 0.91)
- Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding (ICLR 2026 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.91)
- Dynamic-Width Speculative Beam Decoding for LLM Inference (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
