Research map / Large language models / Reasoning and chain of thought

AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

AAAI 2025 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaSkip, a training-free adaptive sublayer (attention/FFN) skipping method that uses on-the-fly IO cosine similarity to skip unimportant sublayers in both prefilling (offline-learned) and decoding (online-learned) for long-context LLM inference.

Problem

Layer-wise skipping methods for LLMs use fixed layers, ignore sublayer (attention vs FFN) importance differences, and do not accelerate the prefilling phase, which hurts generation quality in long-context inference.

Approach

Measure per-sublayer importance by input-output cosine similarity with a modulus scale compensation; learn skip sets offline from historical prefilling tasks and refine FFN skipping online from the first decoded tokens, enabling independent attention/FFN skipping in both phases.

Key results

• On LLaMA3.1-8B-128k with 8 skipped sublayers, AdaSkip reaches 86.6 F1 on TriviaQA and 72.8% accuracy on TREC, near the full model (91.6 and 75.0), while Early Exit, SkipDecode and Unified Skipping degrade sharply.
• In decoding tasks, AdaSkip achieves up to 1.47x actual speedup (InternLM, 16 skipped sublayers) with Rouge-L comparable to the full model.
• In end-to-end skipping (prefill + decode) with 16 skipped sublayers, AdaSkip keeps GovReport Rouge-L at 18.9 on LLaMA3.1-8B-128k, whereas baselines drop below 5.

Measured effects reported in the paper

Surprising findings

• Skipping FFN sublayers is more beneficial in decoding than prefilling because FFN similarity is higher in decoding, while attention sublayers show the opposite trend.
• SkipDecode and Unified Skipping collapse to near-zero scores when layer skipping is applied in the prefilling phase.

Limitations

• Speedup in prefilling for LLaMA is lower than baseline because FFN time dominates after attention parallelism optimizations.
• Evaluation limited to three 7-8B models, a single L20 GPU and a small set of benchmarks.

Experimental setup

Datasets: 2WikiMQA MultiFieldQA-en TriviaQA TREC GovReport MultiNews

Benchmarks: MultiFieldQA-en TriviaQA TREC GovReport MultiNews

Metrics: F1 accuracy Rouge-L theoretical speedup actual speedup GPT score layer hit rate

Baselines: Early Exit SkipDecode Unified Skipping

Builds on: SkipDecode early exit LongBench

Closest papers by meaning

• CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.90)
• Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.89)
• DLP: Dynamic Layerwise Pruning in Large Language Models (ICML 2025 · similarity 0.89)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.89)
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.89)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)