Research map / Large language models / Reasoning and chain of thought

D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives per-position loss weights for parallel block drafters (DFlash) from a differentiable surrogate of expected accepted draft length, giving a detached weighted cross-entropy (D-PACE) that improves emitted length and speedup over DFlash's fixed exponential decay.

Problem

Parallel block drafters for speculative decoding use fixed position-dependent loss weights (e.g., DFlash exponential decay, Medusa head weights) that do not adapt to which positions currently limit accepted draft length.

Approach

Differentiate a smooth surrogate of expected accepted length (sum of cumulative draft-confidence products) with respect to each position's confidence, yielding closed-form per-position weights (cumulative prefix confidence times continuation value) that are detached and applied to token-level CE; asymmetric smoothing of confidences is applied only inside the weights.

Key results

• On Qwen3-4B 3L at T=0, D-PACE average SR 3.20 and tau 4.52 vs DFlash 2.93 and 4.10 across six benchmarks.
• On Llama-3.1-8B-Instruct and Qwen3-8B (3L), D-PACE improves average emitted length by about +12.5% and +12.8%.
• Training-time overhead is 2.3% (177 ms vs 173 ms per step on a single H200).

Surprising findings

• The Accept-rate baseline, which directly optimizes the accepted-length surrogate, collapses to tau near 1.0, performing far worse than DFlash.
• Using target-model probabilities instead of draft probabilities in the weight recovers only a small part of the gain (tau 4.25 vs 4.52).

Limitations

• Derivation applies only to parallel block drafters generating all positions in one forward pass; not to sequential drafters.
• Draft confidence on target-selected token is a heuristic proxy for acceptance, possibly weakly aligned early in training.
• Evaluated only for lossless speculative decoding with hard-match verification.

Experimental setup

Datasets: Qwen3-4B-Instruct-100K ShareGPT Nemotron-Post-Training-Dataset-v2 CodeAlpaca

Benchmarks: GSM8K MATH-500 HumanEval MBPP MT-Bench Alpaca

Metrics: wall-clock speedup (SR) average emitted length (tau)

Baselines: DFlash decayed-CE Top-3 prefix mask Accept-rate

Builds on: DFlash GRIFFIN EAGLE-2 Medusa

Closest papers by meaning

• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.93)
• BASTION: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting (NeurIPS 2026 · similarity 0.91)
• Steering Pretrained Drafters During Speculative Decoding (AAAI 2026 · similarity 0.91)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.91)
• Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.90)
• Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing (NeurIPS 2026 · similarity 0.90)