AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding

Liang He, SiYuan Ma, Qishi Zhan, Yongqi Fan, Mingyu Cao, Qizhen Lan, Zhaolu Kang, Xilu Wang

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ValuSpec, a plug-and-play candidate valuation step that scores draft-tree branches before target-model verification in tree-based speculative decoding.

Problem

Tree-based speculative decoding with retrieval-reused drafts produces candidate trees with many low-quality branches as trees grow, wasting verification compute in serving scenarios with overlapping requests.

Approach

Insert a valuation stage that scores and prunes candidate branches before target verification, so that only valuable candidates are verified; designed to be plug-and-play with existing retrieval-based tree drafting methods.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add ValuSpec pre-verification filtering to Retrieval SDRetrieval SD · Dolly, Llama-2-7B (A100)throughput (tokens/s): 45.68 -> 56.19Improved
add ValuSpec filtering on top of EAGLE-3 dynamic treesEAGLE-3 · GSM8K, Llama-3.1-8B-Instructthroughput (tokens/s): 70.40 -> 98.25Improved
add ValuSpec filtering on top of EAGLE-3 dynamic treesEAGLE-3 · SpecBench, Llama-3.1-8B-Instructthroughput (tokens/s): 47.70 -> 69.17Improved
calibrate threshold on a different dataset (cross-dataset transfer) instead of the same one (ablation)ValuSpec · Llama-2-7B, transfer across HumanEval/GSM8K/Dollyspeedup over AR: 2.05x--2.29x on 7B, 2.07x--2.35x on 13BNo clear effect
replace calibrated threshold with best fixed likelihood threshold (ablation)ValuSpec · Qwen3.5 Retrieval-SD, HumanEvalthroughput (tokens/s): 87.15 -> 113.16Improved
replace calibrated threshold with best fixed likelihood threshold (ablation)ValuSpec · Qwen3.5 Standard SD, HumanEvalthroughput (tokens/s): 58.26 -> 79.78Improved
increase ValuSpec draft length n from 1 to 4 (ablation)ValuSpec · Dolly, Llama-2-7B (A100)throughput (tokens/s): 43.78 -> 56.19Improved
increase ValuSpec draft length n beyond 4 (to 5 and 6) (ablation)ValuSpec · Dolly, Llama-2-7B (A100)throughput (tokens/s): 56.19 -> 49.36 (n=5), 44.25 (n=6)Hurt
increase SD draft length n beyond 3 (ablation)SD · Dolly, Llama-2-7B (A100)throughput (tokens/s): 34.89 -> 33.92 (n=4), 32.41 (n=5)Hurt
increase batch size from 1 to 8 with ValuSpec (ablation)ValuSpec on Retrieval-SD · Qwen3.5 batched inferencespeedup: 2.01 / 1.73 / 1.47x for batch 1 / 4 / 8Hurt
run on RTX 4090 instead of A100 (ablation)ValuSpec · Llama-2-7B, HumanEvalspeedup over AR: 2.31 (A100) vs 1.84 (RTX 4090)Mixed
add ValuSpec filtering to EAGLE-3 trees to raise acceptance of retained candidatesEAGLE-3 · Llama-3.1-8B-Instructacceptance rate: 48.2%-71.4% -> above 86%Improved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Experimental setup

Builds on: speculative decoding tree-based speculative decoding

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy