Research map / Architectures and efficiency / Attention and transformers
What Layers When: Learning to Skip Compute in LLMs with Residual Gates
Filipe Laitenberger, Dawid Jan Kopiczko, Cees G. M. Snoek, Yuki M Asano
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GateSkip token skipping instead of random skipping | Llama-3.2-1B · Generative avg, 15% saved | accuracy: 1.67 -> 22.14 | Improved |
| Use GateSkip instead of MoD router tuning, CALM, FREE, LayerSkip | Llama-3.2-1B · Generative avg, 15% saved | accuracy: MoD 3.96, FREE 10.70, LayerSkip 10.65 -> 22.14 | Improved |
| Place gate after the module output rather than at entry (ablation) | GateSkip · Generative avg, 5% saved | accuracy: 5.5 -> 25.5 | Improved |
| Use vector gates instead of scalar gates (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 20.4 -> 23.2 | Improved |
| Use separate gates per layer instead of shared (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 20.7 -> 23.2 | Improved |
| Skip both attention and MLP rather than only one (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 14.9 (attn only), 7.8 (MLP only) -> 23.2 | Improved |
| Use a linear-sigmoid gate instead of an MLP gate (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 18.5 -> 23.2 | Improved |
| Use L2 gate loss instead of L1 (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 21.4 -> 23.2 | Improved |
| Use L2 loss instead of layer-wise KL regularizer (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 9.8 -> 23.2 | Improved |
| Unfreeze the backbone while training gates (ablation) | GateSkip · Generative avg, 15% saved | accuracy: 12.7 -> 23.2 | Improved |
| Apply GateSkip to instruction-tuned Llama-3B | Llama-3.2-3B-Instruct · Generative avg | accuracy: 36.5 -> 49.0 | Improved |
| Add 4-bit quantization to GateSkip | Llama-3.2-3B · Generative avg, 0% saved | accuracy: 45.0 -> 42.5 | No clear effect |
Closest papers by meaning
- Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.87)
- SkipGPT: Each Token is One of a Kind (ICML 2025 · similarity 0.86)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.86)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.85)
- Attention Drift: What Auto-Regressive Speculative Decoding Models Learn (NeurIPS 2026 · similarity 0.85)
- Masked Gated Linear Unit (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
