Research map / Large language models / Reasoning and chain of thought

RheoSampling: Resolving the One-Hot Dilemma in Stochastic Dynamic-Tree Speculative Decoding

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RheoSampling, which injects one sampled token into EAGLE-3-style dynamic top-K trees using a proxy probability for tree construction and the true sampling probability for OT-based verification, claiming losslessness and higher acceptance length than top-K.

Problem

Dynamic-tree speculative decoding (EAGLE-2/3) is deterministic and collapses the draft distribution to one-hot under stochastic decoding (T>0), causing a large acceptance-rate drop, while static trees keep stochastic sampling but lose context-aware topology.

Approach

Decouple the two roles of a sampled draft token: give it a proxy probability min{q(x_m), z} for expansion/reranking/pruning so it survives into the dynamic tree, and its true residual sampling probability for verification, with an OT-based multi-draft verifier and top-128 sparse drafting.

Key results

• On Llama-3.1-8B-Instruct, average acceptance length rises from 5.04 (Top-K) to 5.25 with RheoSampling, and speedup from 2.84x to 2.93x.
• On DeepSeek-R1-Distill-Llama-8B, average tau improves from 4.99 to 5.21 with speedup 2.89x to 2.99x.
• In a controlled single-layer MT-bench test, RheoVerification reaches 85.4% acceptance vs 83.5% for RRSw and 80.0% for vanilla Top-K.

Surprising findings

• The relative speedup gain (about 3%) is smaller than the relative acceptance-length gain (about 4%) because of the overhead of sampling and verifying the stochastic probe.
• Top-128 sparse truncation of the draft distribution gives acceptance length comparable to the full distribution (4.78-4.85 vs 4.89).

Limitations

• Gains are modest (tau +0.14 to +0.22) and the paper's statistical significance claim is not formally tested.
• Evaluation uses only 80 questions per benchmark on a single GPU type and three target models.
• Proxy design is chosen empirically; the m=1 choice is tuned on the same benchmarks.

Experimental setup

Datasets: Alpaca GSM8K HumanEval MT-bench Natural Questions CNN/DailyMail

Benchmarks: Alpaca GSM8K HumanEval MT-bench Natural Questions CNN/DailyMail

Metrics: average acceptance length wall-clock speedup acceptance rate

Baselines: EAGLE-3 top-K RRSw

Builds on: EAGLE-3 EAGLE-2 SpecInfer SpecTr Sequoia

Closest papers by meaning

• TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.91)
• STree: Speculative Tree Decoding for Hybrid State Space Models (NeurIPS 2025 · similarity 0.91)
• Traversal Verification for Speculative Tree Decoding (NeurIPS 2025 · similarity 0.90)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
• BASTION: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting (NeurIPS 2026 · similarity 0.90)
• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.90)