Research map / Architectures and efficiency / RNNs and normalization
Set-based Neural Network Encoding Without Weight Tying
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SNE, a set-based neural network weight encoder using set-to-set and set-to-vector attention functions with pad-chunk-encode and logit invariance regularization instead of weight tying, evaluated on INR frequency prediction and new cross-dataset and cross-architecture CNN performance prediction tasks.
Problem
Predicting properties (e.g., generalization performance, INR frequency) of neural networks from their weights alone, in a way that works across mixed architectures and parameter sizes without weight tying.
Approach
Encodes weights hierarchically (chunk, layer, network) with set-attention functions and positional encodings for layer type and level, replacing architecture-specific weight tying with a logit invariance regularizer so a single encoder can process arbitrary architectures.
Key results
- On the cross-dataset task, SNE averages Kendall's tau 0.761 versus 0.746 for NFN_NP (second best) and 0.616 for MLP.
- On the INR frequency prediction task, SNE reaches MSE 0.098 versus 7.452 for the encoder without positional and hierarchical encoding (ablation).
- SNE transfers from Arch1 to Arch2 model zoos with Kendall's tau reported per dataset pair, outperforming DeepSets and NeuralGraph baselines according to the authors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SNE instead of NFN_NP | cross-dataset performance prediction · Cross-dataset task | Kendall's τ: 0.746$\pm{0.106}$ -> 0.761$\pm{\textbf{0.101}}$ | Helped |
| add layer-level encoder (ablation) | SNE · INR frequency prediction | MSE: 2.216$\pm{1.303}$ -> 0.098$\pm{\textbf{0.002}}$ | Helped |
| add layer-type encoder (ablation) | SNE · INR frequency prediction | MSE: 0.128$\pm{0.007}$ -> 0.098$\pm{\textbf{0.002}}$ | Helped |
| add set functions (ablation) | SNE · INR frequency prediction | MSE: 1.931$\pm{0.108}$ -> 0.098$\pm{\textbf{0.002}}$ | Helped |
| add positional and hierarchical encoding (ablation) | SNE · INR frequency prediction | MSE: 7.452$\pm{0.799}$ -> 0.098$\pm{\textbf{0.002}}$ | Helped |
| add invariance regularization (ablation) | SNE · INR frequency prediction | MSE: 0.156$\pm{0.015}$ -> 0.098$\pm{\textbf{0.002}}$ | Helped |
| vary chunksize among 4, 8, 16, and 32 (ablation) | SNE · INR frequency prediction | MSE: 0.095 $\pm{0.012}$; 0.090 $\pm{0.023}$; 0.118 $\pm{0.019}$; 0.056 $\pm{0.020}$ | No clear effect |
Surprising findings
- Removing the positional and hierarchical encoding degrades INR MSE from 0.098 to 7.452, indicating the layer-level information is critical.
- Without invariance regularization, performance degrades only mildly on CNN tasks, though the authors report it has a large impact for MLPs such as INRs.
Limitations
- Evaluated only on CNN and INR model zoos; transformer and ResNet results are limited to a single transfer experiment or are discussed only theoretically.
- Predicts generalization performance and INR frequency only; other properties and downstream tasks like weight generation are left to future work.
- Equivariance is only approximately enforced via regularization, relying on a theoretical argument that is stated under linearity assumptions.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR10 SVHN INR model zoo
Benchmarks: INR frequency prediction cross-dataset performance prediction cross-architecture performance prediction transformer model zoo transfer
Metrics: MSE Kendall's tau
Baselines: MLP DeepSets HyperRep STATNN DWSNet NFN_NP NFN_HNP NFT NeuralGraph
Builds on: Set Transformer DeepSets Logit Invariance Regularization Neural Functional Networks DWSNet
Closest papers by meaning
- Encodings for Prediction-based Neural Architecture Search (ICML 2024 · similarity 0.87)
- Towards Scalable and Versatile Weight Space Learning (ICML 2024 · similarity 0.86)
- Distribution Consistent Neural Architecture Search (CVPR 2022 · similarity 0.85)
- Neural Architecture Optimization (NeurIPS 2018 · similarity 0.84)
- Learning Useful Representations of Recurrent Neural Network Weight Matrices (ICML 2024 · similarity 0.84)
- Scalable Neural Network Kernels (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.