Research map / Large language models / Reasoning and chain of thought

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TreeGraft, a training-free multi-drafter framework that jointly builds a shared speculative draft tree from a cheap n-gram drafter and a stronger neural middle drafter, with a distilled lightweight scheduler deciding when to call the middle drafter.

Problem

Tree-based speculative decoding uses a single drafter, forcing a trade-off between draft-tree quality (larger drafter) and drafting latency (smaller drafter).

Approach

Two drafters of different cost share one draft tree: the middle drafter revisits previously unvisited nodes to choose grafting positions, attaches children non-destructively without overwriting existing branches, and a scheduler distilled from an offline value system (via call/skip margin) decides which steps call it.

Key results

• TreeGraft averages 1.60x speedup across 10 model pairs and 6 benchmarks, vs 1.32x for All Small and 1.39x for All Mid, a 15.1% average gain over the better endpoint.
• Maximum individual gain is 26.6% on Qwen3-32B/0.6B with Alpaca (1.38x vs All Mid 1.09x).
• On 4 held-out model pairs TreeGraft averages 1.48x, above All Small (1.32x) and All Mid (1.20x).

Surprising findings

• A direct value-system planner has a smaller offline gap to the oracle but only 0.95x online speedup, versus 1.60x for the distilled margin scheduler.
• Replacing the neural small drafter with a training-free n-gram drafter was necessary because even the smallest neural LLM adds nontrivial forwarding latency.

Limitations

• Relies on a training-free n-gram small drafter; does not combine with trained draft heads such as EAGLE.
• Offline fitting uses an exhaustive enumeration of 2^5 call/skip trajectories, limited to D=5 drafting steps and a specific set of model pairs and datasets.
• Gains are reported on a single hardware/implementation setup; scheduler generalization is tested on only 4 held-out pairs.

Experimental setup

Datasets: GSM8K Alpaca NQ HumanEval CNN/DailyMail

Benchmarks: GSM8K Alpaca NQ HumanEval CNN/DailyMail MT-Bench

Metrics: speedup over autoregressive decoding average accepted length

Baselines: All Small All Mid Cascade Speculative Drafting (CSD) EAGLE-style candidate pool

Builds on: EAGLE Cascade Speculative Drafting TriForce tree attention speculative decoding

Closest papers by meaning

• Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding (ICLR 2026 · similarity 0.93)
• UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding (NeurIPS 2026 · similarity 0.93)
• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.92)
• TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.92)
• BASTION: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting (NeurIPS 2026 · similarity 0.91)
• Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.91)