Research map / Architectures and efficiency / Pruning and efficient tuning

SkipGPT: Each Token is One of a Kind

ICML 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

SkipGPT dynamically prunes LLM attention and MLP modules per token using learned Gumbel-Softmax routers trained in a two-stage scheme (router tuning, then LoRA), removing over 40% of parameters while matching the dense model.

Problem

Static layer pruning of LLMs ignores token-level heterogeneity in required computation and treats attention and MLP as a single unit, and joint router-plus-model training of dynamic pruning is unstable.

Approach

Route each token per module (attention or MLP) with a learned linear router using straight-through Gumbel-Softmax, allowing compute to be allocated across both tokens and modules under a global sparsity target, and train routers first with frozen weights before LoRA recovery.

Key results

• SkipGPT-RT-L on LLaMA2-7B at 25.5% parameter reduction reaches 69.50 average accuracy vs 68.69 for the dense model.
• Router tuning alone (SkipGPT-RT) on LLaMA3.1-8B at 40.2% reduction retains 59.80 average accuracy versus 43.84 or lower for static baselines like ShortGPT.
• SkipGPT-RT-L on LLaMA3.1-8B at 40.3% reduction achieves 70.29 average accuracy vs 72.69 dense, outperforming all static and dynamic baselines under LoRA recovery.

Measured effects reported in the paper

Surprising findings

• Router tuning prunes attention modules more than MLP modules across all sparsity levels, suggesting attention is more redundant.
• Contrary to prior findings, later tokens in a sequence require more attention computation and less MLP computation as context grows.

Limitations

• Parameter reduction is measured as average per-token non-participating parameters, not wall-clock speedup.
• Joint training of router and model underperforms, and the method relies on a sparsity penalty weight (alpha=8) tuned empirically.

Experimental setup

Datasets: RedPajama-Data-1T-Sample WikiText2 PTB BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA

Benchmarks: BoolQ PIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenbookQA WikiText2 PTB

Metrics: accuracy normalized accuracy perplexity

Baselines: ShortGPT Shortened LLaMA LaCo Joint Layer Drop LLM-Pruner SliceGPT MoD-D D-LLM SkipGPT-Joint

Builds on: Gumbel-Softmax Mixture-of-Depths LoRA ShortGPT

Closest papers by meaning

• Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing (ICLR 2025 · similarity 0.91)
• Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.90)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.90)
• SlimGPT: Layer-wise Structured Pruning for Large Language Models (NeurIPS 2024 · similarity 0.90)
• Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization (NeurIPS 2025 · similarity 0.89)
• Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.89)