Research map / Optimization / Optimizers
MetaInit: Initializing learning by learning to initialize
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MetaInit to DeltaOrthogonal initialization | Plain 28-10, no normalization · CIFAR-10 | test error (%): 90.0 -> 3.7 | Helped |
| apply MetaInit to DeltaOrthogonal initialization | WideResnet 202-4 · CIFAR-10 | test error (%): 6.7 -> 3.8 | Helped |
| apply MetaInit to DeltaOrthogonal initialization | WideResnet 28-10 · CIFAR-10 | test error (%): 3.2 -> 2.9 | Helped |
| use MetaInit without normalization instead of Batchnorm | Plain 28-10 · CIFAR-10 | test error (%): 6.0 -> 3.7 | Helped |
| use MetaInit without normalization instead of Batchnorm | WideResnet 202-4 · CIFAR-10 | test error (%): 3.4 -> 3.8 | Hurt |
| use MetaInit instead of LSUV | WideResnet 28-10 · CIFAR-10 | test error (%): 4.8 -> 2.9 | Helped |
| add MetaInit on top of Batchnorm | WideResnet 28-10 · CIFAR-10 | test error (%): 2.8 -> 2.8 | No clear effect |
| apply MetaInit to DeltaOrthogonal with Swish | Resnet-50 · ImageNet top-1 | top-1 test error (%): 99.9 -> 24.0 | Helped |
| apply MetaInit to DeltaOrthogonal with ReLU | Resnet-50 · ImageNet top-1 | top-1 test error (%): 24.3 -> 24.6 | Hurt |
| use L1 norm instead of L2 in gradient quotient (ablation) | MetaInit · 28-layer linear net meta loss | meta loss | Helped |
| use MetaInit instead of signSGD during supervised training to fix bad init (ablation) | WideResnet 16-4 · CIFAR-10 5 epochs | test error (%) | Helped |
Closest papers by meaning
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.89)
- Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (ICLR 2018 · similarity 0.88)
- AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks (AAAI 2023 · similarity 0.88)
- Initialization and Regularization of Factorized Neural Layers (ICLR 2021 · similarity 0.87)
- MetaNorm: Learning to Normalize Few-Shot Batches Across Domains (ICLR 2021 · similarity 0.87)
- Learning To Learn by Jointly Optimizing Neural Architecture and Weights (CVPR 2022 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.