Research map / Architectures and efficiency / RNNs and normalization
DεpS: Delayed ε-Shrinking for Faster Once-For-All Training
Aditya Annavajjala, Alind Khare, Animesh Agrawal, Igor Fedorov, Hugo M Latapie, Myungjin Lee, Alexey Tumanov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| start shrinking at 50% of full-model training instead of 25% or 75% (ablation) | DepS · ImageNet-1k | subnet accuracy: about 1% degradation around 200 MFLOPs at 75% vs 50% | Improved |
| add epsilon-shrinking (incremental subnet learning-rate warmup) (ablation) | DepS · ImageNet-1k full model | full model top-1 accuracy: avoids a 2% accuracy drop at epoch 150 | Improved |
| use IKD-Warmup instead of plain inplace KD (ablation) | DepS · ImageNet-1k | subnet top-1 accuracy: +3.5% at 560 MFLOPs, +2% at 150 MFLOPs | Improved |
| delay shrinking (train full model partially) versus complete full-model training | MobileNet-based supernet initialization · CIFAR-10, CIFAR-100, ImageNet-1k | small subnet accuracy | Improved |
| replace OFA with DepS on ImageNet-1k MobileNetV3 | once-for-all training · ImageNet-1k | training cost (GPU hours): 1256 -> 700 | Improved |
| apply DepS on ProxylessNAS space instead of OFA | once-for-all training · ImageNet-1k (ProxylessNAS space) | smallest subnet accuracy: +1.8% | Improved |
Closest papers by meaning
- LEGACY: A Lightweight Dynamic Gradient Compression Strategy for Distributed Deep Learning (ICLR 2026 · similarity 0.88)
- MEET: Towards Memory-Efficient Temporal Sparse Deep Neural Networks (CVPR 2025 · similarity 0.86)
- LEMON: Lossless model expansion (ICLR 2024 · similarity 0.85)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.85)
- Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training (CVPR 2025 · similarity 0.85)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
