Research map / Large language models / Reasoning and chain of thought

TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TriSpec, a ternary speculative decoding framework that routes verification between a same-family small proxy verifier and the large target model using a top-1/top-2 margin criterion, reducing target invocations and speeding up EAGLE-3/HASS drafting.

Problem

Speculative decoding speedups have saturated on drafting and acceptance, but the per-round target-model verification cost remains a major latency bottleneck for LLM reasoning with long outputs.

Approach

Use a smaller same-family model (e.g., Qwen3-1.7B for Qwen3-32B) as a lightweight proxy verifier that pre-verifies drafts; a top-1 minus top-2 probability margin decides whether the proxy's verification is trusted or whether the draft must be escalated to the target model, so the target is invoked only on uncertain tokens.

Key results

• On Qwen3 at temperature 0, EAGLE3 + TriSpec reaches 3.55x average speedup versus 2.85x for EAGLE3 with about 0.2% average accuracy drop.
• On MATH500 with Qwen3-32B, EAGLE3 + TriSpec cuts target-invocation ratio from 21.38% to 9.95% and per-round verification time from 0.094s to 0.071s.
• Speedups over EAGLE3 on DeepSeek-R1-Distill-Llama-70B reach 3.36x at temperature 0 versus 2.48x for EAGLE3 alone, with 1.6% average accuracy drop.

Surprising findings

• Confidence-filter relaxation on single-layer drafters causes repeated outputs and large accuracy loss, while relaxing verification with same-family smaller models is tolerable.
• Relaxed verification with SpecCascade's draft-distribution-based rules (Chow, OPT) reduces average accuracy by over 20 points on Qwen3 at temperature 0.

Limitations

• Requires training a drafter adapter with hidden features from both proxy and target, adding about 4-5 days of training for 32B targets.
• Proxy must be a same-family smaller model; gains depend on alignment between proxy and target.
• Accuracy drop is small but nonzero, and the margin threshold lambda must be tuned for the accuracy-latency trade-off.

Experimental setup

Datasets: ShareGPT UltraChat-200K

Benchmarks: GSM8K MATH500 Gaokao2023-en HumanEval MBPP SpecBench HotpotQA Polymath AIME24 AIME25 GPQA-Diamond

Metrics: pass@1 accuracy speedup tokens per second target-invocation ratio average acceptance length LLM-judge score

Baselines: EAGLE-3 HASS SpecCascade (Chow) SpecCascade (OPT) SpecCascade (Token) Single model proxy R2R Judge Decoding

Builds on: EAGLE-3 HASS speculative sampling SpecCascade Judge Decoding

Closest papers by meaning

• VIA-SD: Verification via Intra-Model Routing for Speculative Decoding (ICML 2026 · similarity 0.92)
• SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.91)
• AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders (NeurIPS 2025 · similarity 0.91)
• PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.91)
• SafeSpec: Fast and Safe LLM via Dynamic Reflective Sampling (ICML 2026 · similarity 0.90)
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.90)