Research map / Architectures and efficiency / RNNs and normalization
Know What You Need: Efficient Activation Checkpointing
Danyal Ahmed, Edouard Oyallon
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper targets the compute overhead of activation checkpointing for transformer linear layers during LLM training (abstract truncated, so the proposed method is not specified).
Problem
LLM training is memory-bound; activation checkpointing saves memory by recomputing activations in backprop, but rematerializing linear layers of transformers is prohibitively expensive in FLOPs.
Approach
Abstract truncated before the proposed approach is stated; the paper's specific mechanism for efficient checkpointing of linear layers cannot be determined from the provided text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Top-k selection instead of random selection (2/3 kept) (ablation) | KEAP on FFN (Llama-488M) · FineWeb, token-level | perplexity: 12.84 -> 12.23 | Improved |
| use Top-k instead of variance-optimal Opt selection (2/3 kept) (ablation) | KEAP on attention+FFN · FineWeb, token-level | perplexity: 12.81 -> 12.33 | Improved |
| use Opt instead of Rand selection (ablation) | KEAP on FFN · FineWeb, token-level, 1/3 | perplexity: 14.70 -> 13.66 | Improved |
| use token-level instead of batch-level granularity (ablation) | KEAP Top-2/3 on attention+FFN · FineWeb | perplexity: 12.85 -> 12.33 | Improved |
| use KEAP Top-2/3 on FFN instead of compute-matched smaller model | Llama-488M · FineWeb, token-level | perplexity: 12.44 -> 12.23 | Improved |
| use KEAP Top-2/3 on attention instead of compute-matched smaller model | Llama-488M · FineWeb, token-level | perplexity: 12.17 vs 12.17 | No clear effect |
| use batch-level KEAP instead of compute-matched smaller model | Llama-488M · FineWeb, batch-level | perplexity: 12.44 -> 12.61 | Hurt |
| recompute only 2/3 of FFN components (KEAP Top-2/3) | Llama-488M · FineWeb | perplexity: 12.03 -> 12.23 | Hurt |
| reduce kept fraction from 2/3 to 1/3 (Top-k) (ablation) | KEAP on attention+FFN · FineWeb, token-level | perplexity: 12.33 -> 12.90 | Hurt |
| apply KEAP to FFN rather than to attention+FFN (ablation) | KEAP Top-2/3 · FineWeb, token-level | perplexity: 12.33 -> 12.23 | Improved |
| scale to 953M model with Top-2/3 FFN | KEAP · FineWeb | perplexity: 1.66% (488M) -> 1.02% (953M) degradation | Improved |
| batch-level FFN KEAP vs full AC | KEAP Top-1/3 (953M, H100) · training step time | step time: -26% (Top-1/3), -18% (Top-2/3) | Improved |
Closest papers by meaning
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.88)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.87)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.87)
- From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers (AAAI 2025 · similarity 0.87)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.86)
- Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
