Research map / Large language models / Reasoning and chain of thought
AutoJudge: Judge Decoding Without Manual Annotation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoJudge, which automatically mines task-specific important mismatching tokens via semi-greedy search and trains a logistic-regression classifier on LLM hidden states to accept unimportant mismatches during lossy speculative decoding.
Problem
Speculative decoding rejects every draft-target token mismatch, which is overly strict for reasoning and code tasks; prior lossy Judge Decoding needs costly human annotation to label which tokens matter.
Approach
Automatically label mismatching tokens as important or unimportant by semi-greedily replacing target tokens with draft tokens and checking whether the downstream answer (math answer or test outcome) is preserved, then train a lightweight linear classifier on hidden states to override rejections at inference time.
Key results
- On GSM8K with Llama 3.1 70B target, AutoJudge reaches up to about 2x speedup over speculative decoding at at most 1% accuracy drop, accepting over 40 tokens per target forward pass in 8-shot setting.
- On LiveCodeBench, AutoJudge accepts at least 22-25 tokens per speculation cycle at about 2% Pass@1 drop, roughly 2x over speculative decoding.
- In vLLM on A100s, 8B/70B AutoJudge reaches 107.4 tokens/s, about 1.5x over speculative decoding at about 2% accuracy change; offloaded setup up to 2x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoJudge instead of traditional speculative decoding | Llama-3.1-8B draft / Llama-3.1-70B target · GSM8K 8-shot | accepted tokens per target-model forward pass: over 40 tokens | Helped |
| Use AutoJudge instead of traditional speculative decoding | Llama 3.1 70B target · GSM8K | accuracy: ≤1% | Hurt |
| Use AutoJudge instead of traditional speculative decoding | Llama 3.x · LiveCodeBench | accepted tokens per speculative decoding cycle: over 22 tokens | Helped |
| Use AutoJudge instead of traditional speculative decoding | Llama 3.x · LiveCodeBench | Pass@1: ≈2% | Hurt |
| Use AutoJudge instead of speculative decoding | Llama-3.1-8B draft / Llama-3.1-70B target · GSM8K | throughput: 107.4 tokens per second; about 1.5× | Helped |
| Use AutoJudge instead of speculative decoding with an offloaded target | Llama-3.1-8B draft / Llama-3.1-70B target · GSM8K, offloading | speedup: up to 2× | Helped |
| Use AutoJudge instead of standard speculative decoding at equal window size | Llama 3.x · GSM8K, equal window size | speedup: up to 2.5x | Helped |
| Use AutoJudge instead of standard speculative decoding at equal window size | Llama 3.x · GSM8K, equal window size | accuracy: about 2.5% | Hurt |
| Use draft-token embeddings instead of previous-token embeddings (ablation) | AutoJudge classifier · GSM8K validation | classifier accuracy | Helped |
| Use both draft- and target-token embeddings instead of either alone (ablation) | AutoJudge classifier · GSM8K validation | classifier accuracy | Helped |
| Use target-model hidden states instead of draft-model hidden states (ablation) | AutoJudge classifier · GSM8K validation | classifier accuracy | Helped |
| Concatenate draft- and target-model hidden states instead of using target-model states alone (ablation) | AutoJudge classifier · GSM8K validation | classifier accuracy | Helped |
Surprising findings
- Naive per-token important-token mining labels fewer than 1% of tokens as important because a strong target model recovers from individual low-quality tokens, motivating the joint semi-greedy search.
- vLLM greedy inference in bfloat16 gives accuracy that changes noticeably with speculative window size, and half-precision hidden states differ by up to 10% between parallel and sequential passes.
Limitations
- Requires a task-specific equivalence check (answer extraction or test outcome), which is hard for open-ended tasks.
- Running the search consumes substantial inference compute (local or API calls).
- Results on open-ended tasks (creative writing, QA) are weaker; out-of-domain classifiers transfer poorly.
Experimental setup
Datasets: GSM8K LiveCodeBench code_generation_lite release_v5 TriviaQA Arena-Hard-Auto-v2.0 creative writing
Benchmarks: GSM8K LiveCodeBench TriviaQA Arena-Hard-Auto-v2.0 creative writing
Metrics: exact match accuracy accepted tokens per target forward pass Pass@1 tokens per second speedup
Baselines: speculative decoding draft model decoding top-K lossy speculative decoding Judge Decoding EAGLE-2
Builds on: speculative decoding Judge Decoding EAGLE-2 vLLM
Closest papers by meaning
- SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification (ICML 2026 · similarity 0.93)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.92)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.90)
- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (ICLR 2026 · similarity 0.89)
- Reward-Guided Speculative Decoding for Efficient LLM Reasoning (ICML 2025 · similarity 0.89)
- Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.