Research map / Architectures and efficiency / RNNs and normalization
AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks
Irene Tenison, Soumyajit Chatterjee, Fahim Kawsar, Mohammad Malekzadeh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select layers with gradient-free Betti-number ranking instead of Fisher Information / other baselines | pre-trained DNNs (ImageNet) · 16 model-dataset pairs | avg top-1 accuracy (%): 73.75 -> 76.26 (best baseline Last-K vs AdaBet) | Improved |
| AdaBet instead of Fisher Information layer selection | ViT · Stanford Dogs / Oxford Pets / CUB / Flowers102 | top-1 accuracy (%): Stanford Dogs 14.37 -> 85.12; Flowers102 13.10 -> 98.55 | Improved |
| AdaBet instead of ElasticTrainer | ViT · Stanford Dogs | top-1 accuracy (%): 87.79 -> 85.12 | Hurt |
| AdaBet instead of ElasticTrainer | pre-trained DNNs · 16 model-dataset pairs | top-1 accuracy (%): ResNet50 CUB 24.37 -> 46.37; ViT CUB 82.93 -> 79.47 | Mixed |
| train only ~10% layers selected gradient-free vs full training | ResNet50 · peak memory | peak training memory (MB): 777.12 -> 261.78 | Improved |
| AdaBet instead of full training | ViT · peak memory | peak training memory (MB): 386.60 -> 93.64 | Improved |
| AdaBet vs Fisher Information for training time | ResNet50 / ViT · Oxford-IIIT Pets | per-epoch training time: text only, FI shorter than AdaBet | Hurt |
| lower layer proportion rho (ablation) | AdaBet ResNet50 · Stanford Dogs | accuracy and peak memory: memory drops ~65% at rho=0.1 vs 1.0; accuracy stable | Improved |
| lower channel proportion rho_ch to 0.1 (rho=0.5) (ablation) | AdaBet ResNet50 · Oxford-IIIT Pets | accuracy and peak memory: memory drops ~74%; accuracy stable | Improved |
| AdaBet selection step vs ElasticTrainer | ResNet50 / ViT · Oxford-IIIT Pets | selection time: 45% faster | Improved |
| AdaBet vs full training | ResNet50 / ViT · Oxford-IIIT Pets | per-epoch training time: nearly 11% reduction (15% and 7%) | Improved |
Closest papers by meaning
- Adaptive Neural Networks for Efficient Inference (ICML 2017 · similarity 0.85)
- A General Regret Bound of Preconditioned Gradient Method for DNN Training (CVPR 2023 · similarity 0.85)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.85)
- Feature selection and low test error in shallow low-rotation ReLU networks (ICLR 2023 · similarity 0.84)
- AdaShift: Learning Discriminative Self-Gated Neural Feature Activation With an Adaptive Shift Factor (CVPR 2024 · similarity 0.83)
- Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
