AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language Models

Hao Tian, Sheng Lu, Fuwen Tian, Guangming Cui, Zheng Li, Xuyun Zhang, Quan Z. Sheng, Wanchun Dou

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DIAA, a decoding-efficient inference acceleration approach for on-device LLMs built on speculative decoding, targeting the constrained compute and memory of edge devices.

Problem

Speculative decoding for LLM inference is hard to deploy on edge devices because of constrained compute and memory resources.

Approach

Builds on speculative decoding's draft-then-verify paradigm, adapting it to the resource limits of on-device execution; the abstract is truncated so the specific mechanism is not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add LUT-based drafting (ablation)DIAA · Spec-Bench Sumgoodput (tokens/s): 11.91 -> 14.92Improved
add dynamic token tree (ablation)DIAA · Spec-Bench Sumgoodput (tokens/s): 12.62 -> 14.92Improved
add computation overlap (CUDA streams) (ablation)DIAA · Spec-Bench Sumgoodput (tokens/s): 13.60 -> 14.92Improved
add dynamic token tree and LUTs together (ablation)DIAA · Spec-Bench RAGgoodput (tokens/s): 11.34 -> 14.67Improved
dynamic instead of static token treetoken tree drafting · Spec-Bench, Jetson, varying tree sizespeed and acceptance rateImproved
use auxiliary draft model (SpS) instead of model-free draftingspeculative decoding · Llama-3.2-1B-Instruct, Spec-Benchspeedup ratio: 0.94×Hurt
auxiliary draft model adds GPU memorySD with Llama-3.2-1B-Instruct-INT4 draft · Jetson memory usageGPU memory: additional 1 GBHurt
self-drafting by skipping layers (SWIFT)small and mid-size LLMs · Spec-Bench, Vicuna-7b and Llama-3.2-1Bspeedup ratio: 0.97× and 0.93×Hurt
initialize LUTs from a different model with same vocabulary (ablation)DIAA · Spec-Bench Sumgoodput (tokens/s): 14.64 vs 14.38No clear effect
increase tree size (ablation)DIAA · Spec-Benchgoodput and latencyMixed
DIAA vs best baseline (TR) on Llama-3.2-3BDIAA · Spec-Bench averagespeedup ratio: 2.08× -> 2.22×Improved
DIAA vs SAM-D on Vicuna-7bDIAA · Spec-Bench averagespeedup ratio: 1.76× -> 1.78×Improved

Experimental setup

Builds on: speculative decoding

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy