Research map / Architectures and efficiency / RNNs and normalization
Towards Scalable and Versatile Weight Space Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SANE, a sequential transformer autoencoder that learns task-agnostic hyper-representations of neural network weights by tokenizing and encoding weight windows, enabling scaling to ResNet-18 and zero-shot sampling of new models.
Problem
Existing hyper-representation methods embed entire flattened weight vectors at once, limiting them to small networks of fixed architecture and splitting work between discriminative and generative tasks.
Approach
Decompose each model's weights into a token sequence (per-layer, row-wise slices with positional encodings), train the encoder-decoder on random sub-windows of tokens, and embed/sample arbitrarily long models by sequentially processing windows with halo context, so one representation spans architectures and sizes.
Key results
- On CIFAR-10 ResNet-18 sampling at epoch 0, SANE_SUB reaches 68.1% accuracy vs 64.8% for S_KDE30-style SANE_KDE30 and ~10% for random init.
- On ResNet-18 Tiny-ImageNet linear probing, SANE embeddings reach R^2 of 0.795 for accuracy, close to s(W) at 0.802.
- Sampling ResNet-34 for CIFAR-100 from ResNet-18-trained SANE improves 1-epoch accuracy to 16.8% vs 12.4% from scratch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SANE_SUB instead of S_KDE30 sampling | CNN model generation · MNIST, same-task fine-tuning at epoch 0 | accuracy: 68.6±6.7 -> 86.7±0.8 | Helped |
| Use SANE_BOOT instead of SANE_KDE30 sampling | ResNet-18 model generation · CIFAR-10, same-task fine-tuning at epoch 0 | accuracy: 64.8±2.0 -> 68.6±1.2 | Helped |
| Use SANE sampling instead of training from scratch | ResNet-18 transferred from CIFAR-100 to Tiny-Imagenet · Tiny-Imagenet, epoch 1 | accuracy: 10.4±2.2 -> 39.4±1.5 | Helped |
| Add model alignment during pretraining (ablation) | SANE · Model reconstruction, test set | reconstruction loss: 0.167 -> 0.082 | Helped |
| Anchor one contrastive view to the aligned form (ablation) | Aligned SANE pretraining · Model reconstruction, train set | reconstruction loss: 0.148 -> 0.107 | Helped |
| Anchor one contrastive view to the aligned form (ablation) | Aligned SANE pretraining · Model reconstruction, test set | reconstruction loss: 0.082 -> 0.082 | No clear effect |
| Anchor both contrastive views to the aligned form (ablation) | Aligned SANE pretraining · Model reconstruction, train set | reconstruction loss: 0.148 -> 0.072 | Helped |
| Increase inference window size beyond the training window size (ablation) | SANE inference · ResNet sequence reconstruction | reconstruction error | Hurt |
| Add haloing without batch-norm conditioning (ablation) | SANE sampling · CIFAR-10 ResNet-18, epoch 0 | accuracy: 10±0.0 -> 14.5±6.3 | Helped |
| Add batch-norm conditioning without haloing (ablation) | SANE sampling · CIFAR-10 ResNet-18, epoch 0 | accuracy: 10±0.0 -> 60.8±2.2 | Helped |
| Add haloing and batch-norm conditioning together (ablation) | SANE sampling · CIFAR-10 ResNet-18, epoch 0 | accuracy: 10±0.0 -> 64.8±2.1 | Helped |
| Add haloing and batch-norm conditioning together (ablation) | SANE sampling · CIFAR-10 ResNet-18, epoch 5 | accuracy: 90.8±0.2 -> 90.9±0.2 | No clear effect |
Surprising findings
- Deviating from the training window size at inference introduces interference that increases reconstruction error, and window sizes from 32 to 2048 showed little effect on pretraining loss.
Limitations
- Experiments limited to computer vision tasks and homogeneous architecture zoos for pretraining.
- Sampling requires prompt examples; Gaussian-prior bootstrapping is too expensive for large models.
- Comparison with prior hyper-representation work is limited because prior methods cannot scale to ResNets.
Experimental setup
Datasets: MNIST CNN model zoo SVHN CNN model zoo CIFAR-10 CNN model zoo STL-10 CNN model zoo CIFAR-10 ResNet-18 model zoo CIFAR-100 ResNet-18 model zoo Tiny-ImageNet ResNet-18 model zoo ImageNet pytorchcv VGG and ResNet models
Benchmarks: MNIST CNN model zoo SVHN CNN model zoo CIFAR-10 CNN model zoo STL-10 CNN model zoo CIFAR-10 ResNet-18 zoo CIFAR-100 ResNet-18 zoo Tiny-ImageNet ResNet-18 zoo
Metrics: R^2 (property prediction) test accuracy after fine-tuning reconstruction loss
Baselines: hyper-representations (E_c+D) s(W) weight statistics flattened weights W S_KDE30 training from scratch WeightWatcher metrics
Builds on: hyper-representations git re-basin WeightWatcher KDE30 sampling
Closest papers by meaning
- HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories (CVPR 2025 · similarity 0.87)
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.86)
- Learning Useful Representations of Recurrent Neural Network Weight Matrices (ICML 2024 · similarity 0.86)
- Set-based Neural Network Encoding Without Weight Tying (NeurIPS 2024 · similarity 0.86)
- RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks (ICLR 2025 · similarity 0.85)
- Expand-and-Cluster: Parameter Recovery of Neural Networks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.