Research map / Optimization / Optimizers
Magnitude Invariant Parametrizations Improve Hypernetwork Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a magnitude proportionality between hypernetwork inputs and predicted parameters and proposes Magnitude Invariant Parametrizations (L2 input encoding plus additive residual output) that stabilize and speed up hypernetwork training.
Problem
Hypernetworks are hard to train: convergence is slow, sensitive to hyperparameters, and sometimes fails entirely. The paper argues that with standard fully connected architectures, zero biases and standard initialization, predicted parameter scale is proportional to the hypernetwork input scale, cau
Approach
Map hypernetwork inputs to a constant-norm vector via [cos(gamma*pi/2), sin(gamma*pi/2)] so the input magnitude is independent of gamma, and treat hypernetwork outputs as additive changes to independent learnable primary-network parameters (theta = theta0 + h(E(gamma))). This removes the proportionality without extra training or inference cost.
Key results
- On HyperMorph with SGD, default hypernetworks fail to train (Dice 54.3) while MIP reaches 70.5 on OASIS.
- On Bayesian networks with SGD on OxfordFlowers-102, default reaches 1.4% accuracy while MIP reaches 75.4%.
- With Adam on Scale-Space hypernetworks on OASIS, MIP improves Dice from 81.4 to 84.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MIP instead of the default parametrization | Bayesian NN · MNIST test, Adam | accuracy: 98.1 (1.1) -> 99.1 (0.3) | Helped |
| Use MIP instead of the default parametrization | Bayesian NN · MNIST test, SGD | accuracy: 99.2 (0.2) -> 99.0 (0.2) | Hurt |
| Use MIP instead of the default parametrization | Bayesian NN · OxfordFlowers-102 test, Adam | accuracy: 78.1 (1.9) -> 83.2 (0.3) | Helped |
| Use MIP instead of the default parametrization | Bayesian NN · OxfordFlowers-102 test, SGD | accuracy: 1.4 (0.1) -> 75.4 (0.5) | Helped |
| Use MIP instead of the default parametrization | HyperMorph · OASIS test, Adam | Dice: 71.0 (0.3) -> 72.1 (0.3) | Helped |
| Use MIP instead of the default parametrization | HyperMorph · OASIS test, SGD | Dice: 54.3 (0.4) -> 70.5 (0.2) | Helped |
| Use MIP instead of the default parametrization | Scale-Space HN · OASIS test, Adam | Dice: 81.4 (0.3) -> 84.4 (0.6) | Helped |
| Use MIP instead of the default parametrization | Scale-Space HN · OASIS test, SGD | Dice: 75.3 (2.7) -> 78.8 (1.4) | Helped |
| Add constant-norm input encoding (ablation) | MIP hypernetwork · Scale-Space HN on OASIS and Bayesian NN on OxfordFlowers-102 | accuracy / Dice | Helped |
| Add additive output formulation, predicting parameter changes (ablation) | MIP hypernetwork · Scale-Space HN on OASIS and Bayesian NN on OxfordFlowers-102 | accuracy / Dice | Helped |
| Use input encoding and additive output formulation jointly (ablation) | MIP hypernetwork · Scale-Space HN on OASIS and Bayesian NN on OxfordFlowers-102 | accuracy / Dice | Helped |
| Add batch normalization rather than MIP | Scale-Space hypernetwork · OxfordFlowers-102 test | accuracy | Hurt |
Surprising findings
- Batch normalization and layer normalization fail to resolve the proportionality issue; layer normalization in the hypernetwork makes predicted weights independent of the input and diverged under SGD.
- Default hypernetworks with SGD on OxfordFlowers-102 essentially fail to learn (1.4% accuracy).
Limitations
- Experiments use only fully connected hypernetworks with common activation and initialization choices.
- Only SGD with momentum and Adam were tested.
- Focused on training from scratch; transfer learning with pretrained models not studied.
Experimental setup
Datasets: MNIST OxfordFlowers-102 OASIS
Benchmarks: MNIST OxfordFlowers-102 OASIS
Metrics: top-1 accuracy Dice score training loss convergence gradient norm variance
Baselines: default hypernetwork BatchNorm-P LayerNorm-P LayerNorm-H WeightNorm
Builds on: HyperMorph Kaiming initialization weight normalization Adam
Closest papers by meaning
- HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories (CVPR 2025 · similarity 0.88)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.87)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.87)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.87)
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.87)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.