AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Cost-Aware Dynamic Tree Construction for Efficient Large Language Model Inference

Yinrong Hong, Zhiquan Tan, Kai Hu

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a cost-aware dynamic tree construction for speculative decoding that accounts for system-level costs, improving on EAGLE-2 and EAGLE-3 tree structures.

Problem

LLM autoregressive inference is latency-bound; existing dynamic-tree speculative decoding methods (EAGLE-2, EAGLE-3) neglect system-level costs of verifying larger token trees.

Approach

Builds the draft token tree dynamically while explicitly accounting for the system cost (verification/computation cost) of tree size, rather than optimizing only expected acceptance length.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add dynamic reranking (cost-aware choice of nodes to verify) (ablation)EAGLE-3 · ablation, batch 16speedup: 1.35x -> 2.17xImproved
add dynamic reranking (ablation)EAGLE-3 · ablation, batch 1speedup: 3.99x -> 3.99xNo clear effect
add depth pruning on top of DR (ablation)CAST · ablation, batch 8speedup: 2.77x -> 2.84xImproved
add breadth pruning on top of DR and DP (ablation)CAST · ablation, batch 16speedup: 2.27x -> 2.35xImproved
use cost-aware tree instead of EAGLE-3 treeEAGLE-3 · Vicuna-13B HumanEval batch 1 T=0speedup: 4.73x -> 5.18xImproved
use cost-aware tree instead of EAGLE-3 under batchingEAGLE-3 · Vicuna-13B HumanEval batch 8 T=0speedup: 1.91x -> 3.12xImproved
use cost-aware tree vs EAGLE-3 on GSM8K Vicuna-13BEAGLE-3 · Vicuna-13B GSM8K batch 1 T=0speedup: 4.00x -> 3.98xHurt
increase the verified-token limit mEAGLE-3 · accept length vs speedupspeedupHurt
use CAST on other GPUs (H20, RTX 4090)EAGLE-3 · batch 1speedup: 6.46x -> 6.66x (H20, L33 70B HumanEval)Improved
increase batch size to 32EAGLE · HumanEvalspeedup: close to 1xHurt

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Experimental setup

Metrics: inference latency

Baselines: EAGLE-2 EAGLE-3

Builds on: EAGLE-2 EAGLE-3

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy