Research map / Architectures and efficiency / RNNs and normalization
DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Min-K dropping at γ=50% (ablation) | DeiT-Ti training from scratch · ImageNet-1k val | top-1 accuracy: 72.1 -> 72.5 | Helped |
| Use Min-K dropping at γ=70% (ablation) | DeiT-Ti training from scratch · ImageNet-1k val | top-1 accuracy: 72.1 -> 72.1 | No clear effect |
| Use Min-K dropping at γ=80% or γ=90% (ablation) | DeiT-Ti training from scratch · ImageNet-1k val | top-1 accuracy: 72.1 -> 70.8; 72.1 -> 66.4 | Hurt |
| Use random dropping at γ=10% or γ=20% (ablation) | DeiT-Ti training from scratch · ImageNet-1k val | top-1 accuracy: 72.1 -> 72.4 | Helped |
| Use random dropping at γ=30% or more (ablation) | DeiT-Ti training from scratch · ImageNet-1k val | top-1 accuracy: 72.1 -> 72.0 (γ=30%); 72.1 -> 60.8 (γ=90%) | Hurt |
| Drop 90% of applicable activations | DeiT-S · CIFAR-100 | top-1 accuracy: 89.7 -> 90.1 | Helped |
| Drop 90% of applicable activations | DeiT-B · CIFAR-100 | top-1 accuracy: 90.8 -> 91.3 | Helped |
| Drop 90% of applicable activations | Faster R-CNN (ResNet-50-FPN) · COCO | APbox: 37.0 -> 37.2 | Helped |
| Drop 80% of applicable activations | Mask R-CNN (ResNet-50-FPN) · COCO | APbox: 37.9 -> 38.5 | Helped |
| Drop 80% of applicable activations | Mask R-CNN (ResNet-50-FPN) · COCO | APmask: 34.5 -> 34.5 | No clear effect |
| Drop 70% of applicable activations | ResNet-50 · ImageNet-1k | top-1 accuracy: 76.1 -> 76.3 | Helped |
| Apply DropIT to FC layers alongside MESA on other layers | DeiT-S · CIFAR-100 | top-1 accuracy: 89.7 -> 89.9 | Helped |
Closest papers by meaning
- Gradual DropIn of Layers to Train Very Deep Neural Networks (CVPR 2016 · similarity 0.88)
- Dropout with Expectation-linear Regularization (ICLR 2017 · similarity 0.88)
- DropNet: Reducing Neural Network Complexity via Iterative Pruning (ICML 2020 · similarity 0.88)
- Demystifying Dropout (ICML 2019 · similarity 0.87)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.87)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.