Research map / Architectures and efficiency / RNNs and normalization
NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces
Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, Miroslav Pajic
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace flattened/zero-padded vectorization with patch tokenization of weights | NNiT vs D2NWG · PickCube-v1 unseen architectures | success rate: 59% -> 99% | Improved |
| replace flattened/zero-padded vectorization with patch tokenization of weights | NNiT vs D2NWG · StackCubeEasy-v1 unseen architectures | success rate: 42% -> 86% | Improved |
| patch tokenization vs D2NWG on the simple PushCube task | NNiT vs D2NWG · PushCube-v1 unseen architectures | success rate: 89% -> 100% | Improved |
| use SANE row-wise slicing with global positional embeddings on mixed widths | SANE · seen and unseen architectures | success rate: 1% (PickCube seen), 0% (StackCubeEasy) | Hurt |
| generate weights with a GHN with CNN decoder instead of SGD training | training policy dataset · PickCube-v1, 35 seeds | nearest-neighbor L2 distance: 41.73 -> 54.66 | Improved |
| generate weights with a GHN with CNN decoder instead of SGD training | policy dataset · PickCube-v1, 35 seeds | mean success rate: 98.9% -> 99.0% | No clear effect |
| GHN with CNN decoder aligns weight space (vs SGD weights) | weight space structure · PickCube-v1, 35 seeds | qualitative alignment of weight magnitude heatmaps | Improved |
| train jointly on architecture tokens and weight patches (MoNL) for joint generation | NNiT · StackCubeEasy-v1 | success rate: 91% (seen conditional) -> 90% (joint) | Mixed |
| permute weights to remove GHN alignment (ablation) | NNiT · permutation ablation | performance | Hurt |
Closest papers by meaning
- Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion (ICML 2025 · similarity 0.86)
- Graph Metanetworks for Processing Diverse Neural Architectures (ICLR 2024 · similarity 0.85)
- Equivariant Neural Functional Networks for Transformers (ICLR 2025 · similarity 0.85)
- Diffusion-based Neural Network Weights Generation (ICLR 2025 · similarity 0.85)
- $C^2M^3$: Cycle-Consistent Multi-Model Merging (NeurIPS 2024 · similarity 0.85)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
