Research map / Large language models / Reasoning and chain of thought
ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding
Liang He, SiYuan Ma, Qishi Zhan, Yongqi Fan, Mingyu Cao, Qizhen Lan, Zhaolu Kang, Xilu Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ValuSpec, a plug-and-play candidate valuation step that scores draft-tree branches before target-model verification in tree-based speculative decoding.
Problem
Tree-based speculative decoding with retrieval-reused drafts produces candidate trees with many low-quality branches as trees grow, wasting verification compute in serving scenarios with overlapping requests.
Approach
Insert a valuation stage that scores and prunes candidate branches before target verification, so that only valuable candidates are verified; designed to be plug-and-play with existing retrieval-based tree drafting methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ValuSpec pre-verification filtering to Retrieval SD | Retrieval SD · Dolly, Llama-2-7B (A100) | throughput (tokens/s): 45.68 -> 56.19 | Improved |
| add ValuSpec filtering on top of EAGLE-3 dynamic trees | EAGLE-3 · GSM8K, Llama-3.1-8B-Instruct | throughput (tokens/s): 70.40 -> 98.25 | Improved |
| add ValuSpec filtering on top of EAGLE-3 dynamic trees | EAGLE-3 · SpecBench, Llama-3.1-8B-Instruct | throughput (tokens/s): 47.70 -> 69.17 | Improved |
| calibrate threshold on a different dataset (cross-dataset transfer) instead of the same one (ablation) | ValuSpec · Llama-2-7B, transfer across HumanEval/GSM8K/Dolly | speedup over AR: 2.05x--2.29x on 7B, 2.07x--2.35x on 13B | No clear effect |
| replace calibrated threshold with best fixed likelihood threshold (ablation) | ValuSpec · Qwen3.5 Retrieval-SD, HumanEval | throughput (tokens/s): 87.15 -> 113.16 | Improved |
| replace calibrated threshold with best fixed likelihood threshold (ablation) | ValuSpec · Qwen3.5 Standard SD, HumanEval | throughput (tokens/s): 58.26 -> 79.78 | Improved |
| increase ValuSpec draft length n from 1 to 4 (ablation) | ValuSpec · Dolly, Llama-2-7B (A100) | throughput (tokens/s): 43.78 -> 56.19 | Improved |
| increase ValuSpec draft length n beyond 4 (to 5 and 6) (ablation) | ValuSpec · Dolly, Llama-2-7B (A100) | throughput (tokens/s): 56.19 -> 49.36 (n=5), 44.25 (n=6) | Hurt |
| increase SD draft length n beyond 3 (ablation) | SD · Dolly, Llama-2-7B (A100) | throughput (tokens/s): 34.89 -> 33.92 (n=4), 32.41 (n=5) | Hurt |
| increase batch size from 1 to 8 with ValuSpec (ablation) | ValuSpec on Retrieval-SD · Qwen3.5 batched inference | speedup: 2.01 / 1.73 / 1.47x for batch 1 / 4 / 8 | Hurt |
| run on RTX 4090 instead of A100 (ablation) | ValuSpec · Llama-2-7B, HumanEval | speedup over AR: 2.31 (A100) vs 1.84 (RTX 4090) | Mixed |
| add ValuSpec filtering to EAGLE-3 trees to raise acceptance of retained candidates | EAGLE-3 · Llama-3.1-8B-Instruct | acceptance rate: 48.2%-71.4% -> above 86% | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- EAGLE-3 — EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025) · better on 4 of 4 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Experimental setup
Builds on: speculative decoding tree-based speculative decoding
Closest papers by meaning
- TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding (NeurIPS 2026 · similarity 0.90)
- Traversal Verification for Speculative Tree Decoding (NeurIPS 2025 · similarity 0.90)
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.90)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.90)
- UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding (NeurIPS 2026 · similarity 0.89)
- DistillSpec: Improving Speculative Decoding via Knowledge Distillation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
