Research map / Architectures and efficiency / RNNs and normalization
ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
Xinyi. Hu, Yuhao Shen, Zhang Baolin, Hengxin Zhang, Jun Dai, Shuang Ge, Chen Lei, Yue Li, Mingcheng Wan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sparse gating plus elastic budget scheduling instead of static EAGLE-3 trees | EAGLE-3 · Qwen3-235B, BS=1 average | wall-time speedup: 1.69× -> 2.02× | Improved |
| use sparse gating instead of dense per-depth dynamic control (DDD) | DDD · Qwen3-8B, BS=1 average | wall-time speedup: 2.28× -> 2.57× | Improved |
| ECHO vs EAGLE-3 on LLaMA3.3-70B, HumanEval | EAGLE-3 · HumanEval, BS=1 | wall-time speedup: 4.98× -> 5.35× | Improved |
| ECHO vs EAGLE-3 mean accepted tokens on LLaMA3.3-70B HumanEval | EAGLE-3 · HumanEval, BS=1 | mean accepted tokens: 7.12 -> 7.07 | No clear effect |
| sparse gating on small model at high concurrency | EAGLE-3 · LLaMA-3.1-8B, BS=256 | throughput (tokens/s): 10,703 -> 11,551 | Improved |
| elastic budget reallocation across requests | EAGLE-3 · Qwen3-235B, BS=256 | throughput (tokens/s): 2,803 -> 3,207 | Improved |
| use sparse gating instead of dense gating at every depth (ablation) | ECHO · LLaMA-3.1-8B, BS=256 | throughput (tokens/s): 10,978 -> 11,551 | Improved |
| use depth-aware thresholds instead of a fixed threshold (ablation) | ECHO · Qwen3-235B, BS=256 | throughput (tokens/s): 3,046 -> 3,207 | Improved |
| calibrate thresholds on HumanEval and transfer to other datasets (ablation) | ECHO · GSM8K, CNN/DM, Alpaca, MT-Bench | throughput: 1-3% throughput loss | No clear effect |
| raise temperature above 0 with same thresholds (ablation) | ECHO · temperature > 0 | throughput | No clear effect |
| apply ECHO on long context | EAGLE-3 · PG19, BS=16, 16K context | throughput: more than 20% gain | Improved |
| apply ECHO on H20 GPUs | EAGLE-3 · H20 GPUs | throughput: 25% gain | Improved |
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- Speculative Speculative Decoding (ICLR 2026 · similarity 0.91)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.91)
- TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.90)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.90)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
