Research map / Architectures and efficiency / RNNs and normalization
Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GHN-3-XL/m16 predicted parameters instead of random initialization | PyTorch-10 ImageNet models · ImageNet, 1 epoch SGD | top-1 accuracy (%): 25.9±1.3 -> 40.3±3.9 | Helped |
| Initialize with GHN-3-XL/m16 rather than random initialization | ResNet-50 · ImageNet, 1 epoch | top-1 accuracy (%): 20.6 -> 35.0 | Helped |
| Initialize with GHN-3-XL/m16 rather than random initialization | ResNet-50 · ImageNet, 90 epochs | top-1 accuracy (%): 76.0 -> 76.2 | No clear effect |
| Initialize with GHN-3-XL/m16 instead of random initialization | Swin-T · ImageNet, 90 epochs | top-1 accuracy (%): 75.6 -> 76.4 | Helped |
| Use GHN-3-predicted parameters instead of random initialization | ResNet-50 · CIFAR-10 few-shot transfer | accuracy (%): 61.6 -> 72.9 | Helped |
| Fine-tune GHN-3-predicted parameters on ImageNet for 1 epoch before transfer | ResNet-50 · CIFAR-10 few-shot transfer | accuracy (%): 72.9 -> 77.8 | Helped |
| Add predicted-parameter group regularization (ablation) | GHN-3-T/m8 · PyTorch-10, ImageNet, 1 epoch SGD | top-1 accuracy (%): 19.9±4.7 -> 23.9±1.0 | Helped |
| Use group L1-L2 predicted-parameter regularization instead of squared parameter regularization (ablation) | GHN-3-T/m8 · PyTorch-10, ImageNet, 1 epoch SGD | top-1 accuracy (%): 22.5±2.2 -> 23.9±1.0 | Helped |
| Add backward-pass edge embeddings (ablation) | GHN-3-T/m8 · PyTorch-10, ImageNet, 1 epoch SGD | top-1 accuracy (%): 20.4±6.9 -> 23.9±1.0 | Helped |
| Add forward and backward edge embeddings to self-attention (ablation) | GHN-3-T/m8 · PyTorch-10, ImageNet, 1 epoch SGD | top-1 accuracy (%): 18.9±4.1 -> 23.9±1.0 | Helped |
| Add self-attention and edge embeddings instead of an MLP with no edge embeddings (ablation) | GHN-3-T/m8 · PyTorch-10, ImageNet, 1 epoch SGD | top-1 accuracy (%): 13.4±0.8 -> 23.9±1.0 | Helped |
| Increase predicted-parameter regularization strength from 3e-5 to 1e-4 (ablation) | GHN-3-T/m8 · PyTorch-10, ImageNet, 1 epoch SGD | top-1 accuracy (%): 23.9±1.0 -> 20.0±3.0 | Hurt |
Closest papers by meaning
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.86)
- ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks (CVPR 2024 · similarity 0.86)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.85)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.85)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.85)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.