Research map / Architectures and efficiency / RNNs and normalization
On the Angular Update and Hyperparameter Tuning of a Scale-Invariant Network
Juseung Yun, Janghyeon Lee, Hyounguk Shon, Eojindl Yi, Seung Hwan Kim, Junmo Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| raise LR from 0.1 to 3.2 to match tuning factor at N=3125 (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=3125 | test accuracy (%): 71.62 -> 78.28 | Improved |
| lower LR to 0.2 vs 0.1 at N=50000 (matched factor) (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 94.86 -> 95.08 | No clear effect |
| increase LR to 6.4 beyond tuning factor (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 95.13 -> 80.28 | Hurt |
| increase WD to 0.0064 beyond tuning factor (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 95.13 -> 80.18 | Hurt |
| scale LR with batch size (linear scaling) holding N fixed (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 94.87 to 95.13 | No clear effect |
| scale WD linearly with batch size instead of LR (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 95.30 best vs yellow cells | No clear effect |
| scale batch size with N at fixed LR and WD (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 | test accuracy (%): 51.51 -> 79.25 | Improved |
| use tuning factor with SGD (no momentum) (ablation) | SGD scale-invariant ResNet-18 · CIFAR-10 B>512 | test accuracy (%): second-best value not in yellow cell | Mixed |
| use cosine LR schedule (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 95.08 -> 95.14 | No clear effect |
| remove scale-invariant modification (unmodified ResNet-18) (ablation) | SGDM ResNet-18 · CIFAR-10 N=50000 | test accuracy (%): 95.08 -> 95.09 | No clear effect |
| tune WD on 80k subset then rescale by tuning factor | SGDM EfficientNet-B0 · ImageNet | test accuracy (%): best lambda 0.0016 -> predicted 1e-04 vs best 5e-05 | Improved |
| reduce batch size from 128 to 64 at N=3125 (fixed LR WD) (ablation) | SGDM scale-invariant ResNet-18 · CIFAR-10 N=3125 | test accuracy (%): 77.57 -> 79.25 | Improved |
Closest papers by meaning
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.88)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.87)
- The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks (NeurIPS 2024 · similarity 0.87)
- Magnitude Invariant Parametrizations Improve Hypernetwork Learning (ICLR 2024 · similarity 0.86)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
