Research map / Large language models / Reasoning and chain of thought

PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PARD, a target-independent speculative decoding method where one parallel-draft model (mask-token prediction of multiple future tokens in one pass) serves a whole model family, plus conditional drop-token training that cuts training cost ~3x.

Problem

Speculative decoding draft models are usually tied to a specific target (EAGLE-style heads need per-target training), and autoregressive drafting requires K sequential draft passes, limiting LLM inference speed.

Approach

Adapt a small autoregressive draft model into a parallel drafter using mask tokens so all K candidates come from one forward pass, and use conditional drop-token (COD) training, which retains tokens whose prefix KV states are complete, to reduce the K-fold training cost. The draft is independent of target model, so one draft serves a whole family.

Key results

• On LLaMA3.1-8B under vLLM (A100-40GB), PARD reaches 264.88 tokens/s on HumanEval (3.63x speedup) and 1.15x faster than EAGLE-3 on that task.
• Conditional drop-token training is reported to speed up draft training about 3x while keeping inference acceleration; training cost estimated 7x lower than EAGLE and 10x lower than EAGLE-3 for LLaMA3-70B targets.
• PARD's first-token acceptance on HumanEval is 0.93 vs 0.87 for EAGLE-3 with LLaMA3.1-8B target.

Surprising findings

• Using the same mask token ID at all predicted positions outperformed distinct IDs (221.97 vs 218.05 tokens/s) and enables inference with K larger than training K.

Limitations

• Speedups are reported on a single hardware setup (A100-40GB) and a limited set of tasks (code and math).
• Training is done on 8x MI250X and the comparison of training cost relies on analytic FLOP estimates rather than measured wall-clock time.
• Baseline EAGLE numbers are from official models; comparisons under Transformers framework are less favorable to the draft-based baselines.

Experimental setup

Datasets: Magpie-Llama-3.1-Pro-1M Magpie-Qwen2-Pro-1M Evol-CodeAlpaca OpenR1-Math-220k OpenThoughts-114k Chinese-DeepSeek-R1-Distill-Data-110k

Benchmarks: HumanEval GSM8K MATH500 SpecBench

Metrics: tokens per second speedup over autoregressive acceptance rate memory bandwidth consumption training Pflops per 1M tokens

Baselines: AR VSD EAGLE EAGLE-3

Builds on: vanilla speculative decoding Mask-Predict EAGLE-3 HASS

Closest papers by meaning

• Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation (NeurIPS 2026 · similarity 0.91)
• Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.91)
• Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.91)
• Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.90)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.90)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.90)