Research map / Large language models / Reasoning and chain of thought
D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-LLM, which adds per-layer LoRA-trained decision modules that skip transformer layers per token, plus a KV-cache eviction mask, to cut inference FLOPs by about 50% with comparable accuracy on LLaMA2-7B and LLaMA3-8B.
Problem
LLM inference is expensive and every token gets the same computation, which makes deployment on resource-constrained platforms hard; existing static pruning and dynamic methods either degrade accuracy or break KV-cache compatibility.
Approach
Attach a Gumbel-softmax decision module before each transformer layer that decides per token whether to execute or skip that layer, with a loss term that steers average acceleration toward a user-set target, and evict skipped tokens' keys and values from later attention via a sparse mask.
Key results
- D-LLM with about 50% FLOPs matches or exceeds the LoRA baseline on several tasks, e.g., OBQA 0.80 accuracy at 0.53 FLOPs versus 0.79 at 1.00 for LoRA on LLaMA2-7B.
- Reports up to about 45% reduction in KV-cache storage versus the baselines on LLaMA2-7B and LLaMA3-8B.
- On MaWPS, D-LLM reaches 0.74 accuracy at 0.56 FLOPs, versus 0.52 for Shortened-LLaMA (Taylor) at 0.66 FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use D-LLM token-adaptive layer skipping instead of Shortened-LLaMA (Taylor) | LLaMA2-7B · MaWPS | accuracy: 0.39 -> 0.74 | Helped |
| use D-LLM token-adaptive layer skipping instead of Shortened-LLaMA (Taylor) | LLaMA2-7B · GSM8K | accuracy: 0.18 -> 0.29 | Helped |
| use D-LLM token-adaptive layer skipping instead of Shortened-LLaMA (Taylor) | LLaMA2-7B · PIQA | accuracy: 0.83 -> 0.84 | Helped |
| use D-LLM token-adaptive layer skipping instead of Shortened-LLaMA (Taylor) | LLaMA2-7B · Alpaca | PPL: 7.65 -> 6.01 | Helped |
| use D-LLM token-adaptive layer skipping instead of Shortened-LLaMA (Taylor) | LLaMA2-7B · MMLU | accuracy: 0.53 -> 0.53 | No clear effect |
| add KV-cache eviction strategy (ablation) | D-LLM on LLaMA2-7B · Alpaca | PPL: 6.08 -> 6.01 | Helped |
| add KV-cache eviction strategy (ablation) | D-LLM on LLaMA2-7B · SAMSum | PPL: 3.63 -> 3.18 | Helped |
| add KV-cache eviction strategy (ablation) | D-LLM on LLaMA2-7B · MaWPS | accuracy: 0.72 -> 0.74 | Helped |
| add KV-cache eviction strategy (ablation) | D-LLM on LLaMA2-7B · OBQA | accuracy: 0.82 -> 0.80 | Hurt |
| vary reserved initial tokens from m = 0 to m = 2 (ablation) | D-LLM · SAMSum | PPL: 3.33 -> 3.18 | Helped |
| vary reserved initial tokens from m = 0 to m = 2 (ablation) | D-LLM · SIQA | accuracy: 0.80 -> 0.82 | Helped |
| increase loss weight α from 1 to 10 (ablation) | D-LLM on LLaMA2-7B · MaWPS | accuracy: 0.74 -> 0.62 | Hurt |
Surprising findings
- On MaWPS, D-LLM at about 40% FLOPs exceeds the full-cost LoRA baseline, suggesting skipping layers can help on some tasks.
- Layers are skipped mostly in the upper and lower parts of the network, and at very low budgets (~25% FLOPs) most tokens execute only the first three layers.
Limitations
- Generalizability to broader tasks is not yet verified.
- Only explored granularity at the token and layer level; sentence or task-level dynamics are left to future work.
- Performance can degrade with overly large loss weight alpha, and some tasks (e.g., Alpaca) trail the LoRA baseline.
Experimental setup
Datasets: Alpaca SAMSum GSM8K MaWPS BoolQ PIQA SIQA OBQA MMLU
Benchmarks: Alpaca SAMSum GSM8K MaWPS BoolQ PIQA SIQA OBQA MMLU
Metrics: perplexity accuracy FLOPs ratio KV-cache storage
Baselines: LLaMA2-7B + LoRA Mixture-of-Depths Shortened-LLaMA (Taylor) Shortened-LLaMA (PPL) Ada-Infer
Builds on: LoRA Mixture-of-Depths Shortened-LLaMA Ada-Infer Gumbel-Softmax
Closest papers by meaning
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.90)
- Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling (NeurIPS 2025 · similarity 0.89)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.89)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.88)
- Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models (ICLR 2026 · similarity 0.88)
- DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.