Research map / Large language models / Reasoning and chain of thought
DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models
Hao Tian, Sheng Lu, Fuwen Tian, Guangming Cui, Zheng Li, Xuyun Zhang, Quan Z. Sheng, Wanchun Dou
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIAA, a decoding-efficient inference acceleration approach for on-device LLMs built on speculative decoding, targeting the constrained compute and memory of edge devices.
Problem
Speculative decoding for LLM inference is hard to deploy on edge devices because of constrained compute and memory resources.
Approach
Builds on speculative decoding's draft-then-verify paradigm, adapting it to the resource limits of on-device execution; the abstract is truncated so the specific mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LUT-based drafting (ablation) | DIAA · Spec-Bench Sum | goodput (tokens/s): 11.91 -> 14.92 | Improved |
| add dynamic token tree (ablation) | DIAA · Spec-Bench Sum | goodput (tokens/s): 12.62 -> 14.92 | Improved |
| add computation overlap (CUDA streams) (ablation) | DIAA · Spec-Bench Sum | goodput (tokens/s): 13.60 -> 14.92 | Improved |
| add dynamic token tree and LUTs together (ablation) | DIAA · Spec-Bench RAG | goodput (tokens/s): 11.34 -> 14.67 | Improved |
| dynamic instead of static token tree | token tree drafting · Spec-Bench, Jetson, varying tree size | speed and acceptance rate | Improved |
| use auxiliary draft model (SpS) instead of model-free drafting | speculative decoding · Llama-3.2-1B-Instruct, Spec-Bench | speedup ratio: 0.94× | Hurt |
| auxiliary draft model adds GPU memory | SD with Llama-3.2-1B-Instruct-INT4 draft · Jetson memory usage | GPU memory: additional 1 GB | Hurt |
| self-drafting by skipping layers (SWIFT) | small and mid-size LLMs · Spec-Bench, Vicuna-7b and Llama-3.2-1B | speedup ratio: 0.97× and 0.93× | Hurt |
| initialize LUTs from a different model with same vocabulary (ablation) | DIAA · Spec-Bench Sum | goodput (tokens/s): 14.64 vs 14.38 | No clear effect |
| increase tree size (ablation) | DIAA · Spec-Bench | goodput and latency | Mixed |
| DIAA vs best baseline (TR) on Llama-3.2-3B | DIAA · Spec-Bench average | speedup ratio: 2.08× -> 2.22× | Improved |
| DIAA vs SAM-D on Vicuna-7b | DIAA · Spec-Bench average | speedup ratio: 1.76× -> 1.78× | Improved |
Experimental setup
Builds on: speculative decoding
Closest papers by meaning
- SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.91)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.90)
- Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.90)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.90)
- Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.90)
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
