Research map / Learning theory and algorithms / Prediction and decision losses
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| factorize all convolutions and linear layers of ResNet-50 (ablation) | ResNet-50 inference-style forward pass · A100 microbenchmark | runtime: decreases FLOPs significantly but increases runtime | Hurt |
| remove batch normalization ops (ablation) | ResNet-50 · A100 microbenchmark | runtime: reduces time significantly with almost no FLOP change | Helped |
| sample-based approximation of matrix multiplications and convolutions (dynamic row/column pruning) | ImageNet training · ImageNet | training speed: 1.37x faster with little impact on final test accuracy | Helped |
| gradient-norm-based gradual layer freezing (PipeTransformer) | Transformer training · ImageNet/NLP training (as reported) | speedup: up to 2.83x speedup without losing accuracy | Helped |
| data pruning via proxy-model-based selection (SVP) | CIFAR-10 training data · CIFAR-10 | final accuracy: prunes 50% of data without harming final accuracy | Helped |
| k-means embedding-based label-free data pruning on ImageNet | ImageNet training · ImageNet | training samples / accuracy: 20% fewer samples with little accuracy loss | Helped |
| curriculum learning with scoring and ordering choices (ablation) | CIFAR-10, CIFAR-100, FOOD-101(N) · CIFAR-10/100, FOOD-101N | model quality at convergence | No clear effect |
| curriculum learning with dynamic pacing | training speed · multiple tasks (as reported in survey) | training speed: small but significant benefit; benefits notable only with limited iteration budget or noisy labels | Mixed |
| Switch Transformer conditional parameter retrofit vs T5 | T5 transformer · language modeling (as reported) | training speed: up to a trillion parameters, seven times faster than T5 | Helped |
| progressive ViT growing (Growing Ticket Hypothesis / AutoProg) | ViT ImageNet training · ImageNet | training speed: 40-85% speedup with no accuracy drop | Helped |
| stochastic layer dropping (Stochastic Depth/LayerDrop) | ResNet blocks / transformer layers · as reported in survey | training iterations and per-iteration time | Helped |
| tying input and output embedding matrices | language models · language modeling (as reported) | perplexity per training iteration | Helped |
Closest papers by meaning
- DeepOBS: A Deep Learning Optimizer Benchmark Suite (ICLR 2019 · similarity 0.86)
- Learning Gradient Descent: Better Generalization and Longer Horizons (ICML 2017 · similarity 0.86)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.86)
- Approximation and Learning with Deep Convolutional Models: a Kernel Perspective (ICLR 2022 · similarity 0.85)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.83)
- Scaling Down Deep Learning with MNIST-1D (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.