Research map / Large language models / Reasoning and chain of thought

Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FLy, a training-free loosely speculative decoding method that accepts semantically valid draft-target mismatches using a target-entropy gate and a look-ahead deferred window, plus parameter-free prompt-lookup acceleration of the drafter.

Problem

Standard speculative decoding uses strict exact-match verification, rejecting many semantically valid draft tokens and limiting speedup; trained loose variants (e.g., JudgeDecoding, EAGLE) generalize poorly out of distribution.

Approach

A two-tier verification: an entropy gate rejects mismatches at near-deterministic target positions, while at high-entropy mismatches a deferred window of W tokens accepts the mismatch only if the target does not diverge again (no self-correction); drafter is additionally accelerated by prompt lookup decoding (multi-level acceleration).

Key results

• On OOD datasets with Llama-3.3-70B target, FLy achieves 2.53x average speedup vs EAGLE-3's 1.56x (temperature 0 setting reported as 1.62x gap).
• With Llama-3.1-405B target on OOD tasks, FLy reaches 4.80x average speedup (5.21x at temperature 1).
• FLy preserves at least 99% of target accuracy (recovery ratio) across datasets and model scales while requiring no training.

Surprising findings

• Training-based EAGLE-3 achieves 3.83x on in-distribution data but drops to 1.56x on OOD data, while the training-free FLy stays stable.
• Disabling the entropy gate (theta=0) yields higher speedup but reduces recovery to 97.7%, showing the gate is needed for accuracy.

Limitations

• Trails heavily optimized EAGLE-3 on in-distribution benchmarks (e.g., 2.63x vs 3.83x on Llama-3.3-70B).
• Relaxed verification is unsuitable for tasks requiring verbatim reproduction.
• Hyperparameters (W, theta, K) are tuned per setup and the window rule is conservative near draft boundaries.

Experimental setup

Datasets: ACP-prog-gen NIAH-multivalue MGSM GSM8K HumanEval MBPP

Benchmarks: ACP-prog-gen NIAH-multivalue MGSM GSM8K HumanEval MBPP

Metrics: speedup mean accepted tokens (tau) recovery ratio (%)

Baselines: EAGLE-2 EAGLE-3 Speculative Sampling REST TokenRecycling JudgeDecoding

Builds on: speculative sampling EAGLE-3 Prompt Lookup Decoding JudgeDecoding

Closest papers by meaning

• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.92)
• Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs (ICLR 2026 · similarity 0.91)
• SEED: Self-Speculative Decoding via Implicit Encoder–Decoder (NeurIPS 2026 · similarity 0.91)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
• Online Speculative Decoding (ICML 2024 · similarity 0.91)
• GRIFFIN: Effective Token Alignment for Faster Speculative Decoding (NeurIPS 2025 · similarity 0.91)