Research map / Large language models / Language, safety and interpretability

Accelerating Blockwise Parallel Language Models with Draft Refinement

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes block drafts of blockwise parallel LMs and improves them by lattice rescoring of top-k head predictions with small neural LMs or n-gram LMs, yielding up to ~3x wall-clock speedup on Vicuna 7B/13B with Medusa heads.

Problem

Blockwise parallel decoding (BPD) drafts often contain unnatural consecutive repetitions and low-confidence predictions at later heads, limiting the mean accepted token length and inference speedup.

Approach

Treat the top-k predictions at each draft head as a sausage lattice and rescore it without changing base model weights: either greedy local rescoring with a small neural LM, or global rescoring with an n-gram LM via finite-state composition to produce p-best drafts for parallel verification.

Key results

• Local neural rescoring over Medusa on Vicuna 13B improves speedup on MT-Bench from 2.844 to 2.981 at temperature 1.0 and up to 3x wall-clock speedup in greedy settings.
• Global 16-best 4-gram rescoring raises block efficiency on NewsRoom from 1.08 to 1.31 (+21.30%) relative to vanilla BPD.
• Lattice rescoring on Vicuna 7B Medusa improves speedup on MT-Bench at temperature 0.0 from 2.430 to 2.482 and on S-Sum from 2.002 to 2.076.

Measured effects reported in the paper

Surprising findings

• In-domain-trained rescorers performed no better than rescorers trained only on C4.
• Block efficiency gains do not always translate into equivalent wall-clock speedup, attributed to system-level factors.

Limitations

• Gains are task-dependent; little or no improvement on tasks with already high block efficiency such as LAMBADA and CNN/Daily.
• Drafting heads follow the original BPD design and leave room for architectural improvements; the block structure limits rescoring gains.
• Block efficiency improvements do not always translate proportionally into wall-clock speedup.

Experimental setup

Datasets: C4 LAMBADA SQuAD V1 CNN/DailyMail SAMSum MultiNews XSUM NewsRoom ShareGPT

Benchmarks: MT-Bench CNN/DailyMail (S-Sum) SQuAD V1 (QA) GSM8K RAG LAMBADA SAMSum MultiNews XSUM NewsRoom

Metrics: block efficiency speedup ratio perplexity exact match ROUGE-L

Baselines: BPD Medusa Sps Lookahead PLD

Builds on: blockwise parallel decoding Medusa speculative decoding

Closest papers by meaning

• Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.90)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding (NeurIPS 2025 · similarity 0.89)
• DFlash: Block Diffusion for Flash Speculative Decoding (ICML 2026 · similarity 0.89)
• Block Verification Accelerates Speculative Decoding (ICLR 2025 · similarity 0.88)
• PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation (ICLR 2026 · similarity 0.88)
• Learning to Keep a Promise: Scaling Language Model Decoding Parallelism with Learned Asynchronous Decoding (ICML 2025 · similarity 0.88)