Research map / Large language models / Reasoning and chain of thought

SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SelfJudge, which trains a logistic-regression judge verifier for speculative decoding using self-supervised labels from the target model's likelihood changes under token substitution, removing the need for human labels or verifiable ground truth.

Problem

Judge verification for speculative decoding needs token-level acceptance labels, which existing methods obtain via human annotation or answer-preservation on verifiable math/code tasks, limiting generalization to open-ended NLP tasks.

Approach

Label draft tokens as acceptable when substituting them into the target model's own response changes the target's likelihood of the response (prefix plus N suffix tokens) only minimally, yielding a semantic preservation score; a linear verifier on target hidden states is trained on these labels and combined with standard rejection sampling.

Key results

• On Llama-3.1-8B/3.2-1B, SelfJudge-F reaches 11.29 accepted length on GSM8K at 80.5% accuracy (SD: 9.14 / 80.7) and 6.38 on MMLU at 62.7% (SD: 4.36 / 65.0).
• Wall-clock throughput with vLLM on A100: SelfJudge-F gives 137.37 vs 111.21 tokens/sec for SD on GSM8K (+23.5%) and 89.45 vs 61.13 on MMLU (+46.3%).
• On Llama-3.1-70B/8B, SelfJudge-F yields 11.663 accepted length at 94.0% on GSM8K (SD: 9.532 / 93.0%).

Surprising findings

• Judge-only verification yields very low accepted length (e.g., 2.08 on GSM8K), so the judge is useful only combined with alignment-based verification.
• The self-supervised verifier matches or beats the answer-preservation AutoJudge on GSM8K even though it does not use ground-truth answers.

Limitations

• Verifier is a simple linear logistic regression; more complex architectures and scaling laws are not explored.
• Threshold tau is calibrated on GSM8K using AutoJudge labels, and relaxed verification gives small accuracy drops on some tasks (e.g., -2.3 on MMLU for SelfJudge-F).
• Experiments limited to Llama and Qwen families with one draft model per target.

Experimental setup

Datasets: GSM8K LiveCodeBench Dolly15k CNN/DailyMail MATH-500 MMLU

Benchmarks: GSM8K MATH-500 MMLU CNN/DailyMail LiveCodeBench

Metrics: average accepted length accuracy Pass@1 factual consistency score tokens/sec

Baselines: Standard SD Top-k verification AutoJudge-R AutoJudge-F AutoJudge+

Builds on: JudgeDecoding AutoJudge speculative sampling

Closest papers by meaning

• AutoJudge: Judge Decoding Without Manual Annotation (NeurIPS 2025 · similarity 0.93)
• Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.92)
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.91)
• SEED: Self-Speculative Decoding via Implicit Encoder–Decoder (NeurIPS 2026 · similarity 0.90)
• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.90)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.90)