Research map / Large language models / Reasoning and chain of thought

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Presents SpecForge, an open-source framework for training EAGLE-3 speculative decoding draft models with target-draft decoupling and optimized TTT attention kernels, plus SpecBundle, a suite of released draft models.

Problem

Speculative decoding adoption is limited by scarce high-quality EAGLE-3 draft models and by inefficient, monolithic training infrastructure that does not scale to large target models.

Approach

Decouple the frozen target model (run on SGLang with inference-style tensor/expert parallelism) from the lightweight draft model (FSDP/ZeRO-2 training), and reduce TTT memory with FlexAttention BlockMasks and an in-place Triton log-softmax backward.

Key results

• SpecForge achieves up to 9.99x training throughput speedup over the SafeAILab EAGLE-3 baseline for Qwen3-235B-A22B on 8 H200 GPUs.
• Its optimized TTT attention reduces attention kernel time by 62.1% and peak memory by 93.5% at sequence length 4096 on a single H200.
• SpecBundle draft models reach up to 4.48x inference speedup over no speculative decoding (Llama-4-Scout, LiveCodeBench) and up to 1.35x over existing draft checkpoints.

Surprising findings

• MoE draft models consistently underperform dense draft models of comparable parameter or FLOP budgets.
• Data regeneration with the target model improves acceptance length on nearly all benchmarks, contrary to prior claims that EAGLE is insensitive to training data.

Limitations

• Current SpecBundle release focuses on instruct models; reasoning and vision-language models are not yet covered.
• Speedup gains are not uniform; slight MT-Bench decrease for Llama-3 8B and 70B drafts.
• Dynamic TTT length strategies are left to future work.

Experimental setup

Datasets: Open-PerfectBlend ShareGPT UltraChat

Benchmarks: MTBench Math500 GSM8K HumanEval LiveCodeBench GPQA FinanceQA

Metrics: training throughput (tokens/s) speedup inference throughput acceptance length peak memory

Baselines: SafeAILab EAGLE-3 official implementation Existing open-source EAGLE-3 draft checkpoints Hugging Face backend Custom backend

Builds on: EAGLE-3 EAGLE Medusa FlexAttention SGLang

Closest papers by meaning

• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.93)
• SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
• Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.92)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.92)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.92)
• SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.92)