Research map / Architectures and efficiency / RNNs and normalization
Efficient Rematerialization for Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TWRemat instead of no rematerialization | Transformer Base training graph · Transformer Base | peak memory (GiB): 3.97 -> 1.14 | Helped |
| use TWRemat instead of no rematerialization | Transformer Big training graph · Transformer Big | peak memory (GiB): 13.25 -> 2.89 | Helped |
| use TWRemat instead of GreedyRemat | Transformer Base · Transformer Base | peak memory (GiB): 2.92 -> 1.14 | Helped |
| use TWRemat instead of NoRemat | Transformer Base · Transformer Base | relative schedule length: 10.61 | Hurt |
| use TWRemat instead of NoRemat | ResNet and feed-forward networks · depth sweep | peak memory: up to 10x | Helped |
| use GreedyRemat instead of NoRemat | ResNet models · ResNet depth sweep | peak memory: about 2x | Helped |
| use GreedyRemat instead of NoRemat | feed-forward network · FFN depth sweep | peak memory | No clear effect |
| stop TWRemat recursion early (TWRemat(k) with larger k) (ablation) | TWRemat · ResNet200 | peak memory vs schedule length | Mixed |
Closest papers by meaning
- Efficient Combination of Rematerialization and Offloading for Training DNNs (NeurIPS 2021 · similarity 0.89)
- Moccasin: Efficient Tensor Rematerialization for Neural Networks (ICML 2023 · similarity 0.88)
- HiRemate: Hierarchical Approach for Efficient Re-materialization of Neural Networks (ICML 2025 · similarity 0.87)
- Coop: Memory is not a Commodity (NeurIPS 2023 · similarity 0.87)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.85)
- DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.