Research map / Graph learning / GNN architectures
Graph Metanetworks for Processing Diverse Neural Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Graph Metanetworks (GMNs) that represent input neural networks as parameter graphs (one edge per parameter) and process them with message-passing GNNs, giving equivariance to neural DAG automorphisms across diverse architectures.
Problem
Designing metanetworks (neural networks that take other networks' weights as input) that respect parameter permutation symmetries while handling diverse architectures such as attention, normalization, convolution, and residual layers, which prior equivariant metanets cannot process.
Approach
Encode any feedforward network as a compact parameter graph in which each parameter is a single edge (with bias nodes and parameter-sharing handled without scaling with activations), then apply a GNN; neural DAG automorphisms of the graph correspond to permutation symmetries, so the GNN is equivariant to them.
Key results
- On Diverse Architectures with 50% training data, GMN reaches R^2 .975 and tau .908 versus DMC .957/.883 and DeepSets .562/.559.
- With only 5% training data on Diverse Architectures OOD split, GMN achieves R^2 .768 and tau .780 while DMC attains R^2 -.134.
- On self-supervised INR representation learning, GMN reaches test MSE 1.13 versus 1.39 for DWSNets; on Contrast editing GMN gets .0197 MSE, best among methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GMN instead of DMC | CIFAR-10 test accuracy prediction · Varying CNNs, 50% training | R²: .948$\pm.009$ -> \bf .978$\pm.002$ | Helped |
| Use GMN instead of DMC | CIFAR-10 test accuracy prediction · Diverse Architectures, 50% training | R²: .957$\pm.009$ -> \bf .975$\pm .002$ | Helped |
| Use GMN instead of DMC | CIFAR-10 test accuracy prediction · Varying CNNs, 5% training | R²: .816$\pm.038$ -> \bf .876$\pm.010$ | Helped |
| Use GMN instead of DMC | CIFAR-10 test accuracy prediction · Diverse Architectures, 5% training | R²: .810$\pm.046$ -> \bf .918$\pm.002$ | Helped |
| Use GMN instead of DeepSets | CIFAR-10 test accuracy prediction · Varying CNNs, OOD | R²: .741$\pm.015$ -> \bf .891$\pm.037$ | Helped |
| Use GMN instead of DeepSets | CIFAR-10 test accuracy prediction · Diverse Architectures, OOD | R²: .128$\pm.071$ -> \bf .768$\pm.063$ | Helped |
| Use GMN instead of NFT | 2D INR editing · Contrast | Test MSE: .0200 $\pm.0002$ -> \bf .0197 $\pm.0000$ | Helped |
| Use GMN instead of NFT | 2D INR editing · Dilate | Test MSE: \bf .0510 $\pm.0004$ -> .0603 $\pm.0010$ | Hurt |
| Use GMN instead of DWSNets | Self-supervised INR representations · Sinusoid coefficient prediction | Test MSE: 1.39 $\pm .06$ -> \bf 1.13 $\pm.08$ | Helped |
Surprising findings
- On the Dilate INR editing task, GMN is beaten by NFT (.0603 vs .0510 MSE).
- The DeepSets baseline performs poorly on Diverse Architectures OOD (R^2 .128), showing low expressive power.
Limitations
- Not tested on billion-parameter networks; scalability to large models unverified.
- Theory of neural DAG automorphisms is proven for computation graphs; parameter graph extension is only conjectured.
- Only accounts for permutation symmetries, not scaling symmetries of ReLU networks.
Experimental setup
Datasets: CIFAR-10 2D INR editing dataset sinusoid MLP dataset for self-supervised learning
Benchmarks: CIFAR-10 accuracy prediction (Varying CNNs, Diverse Architectures) 2D INR editing (Contrast, Dilate) sinusoid INR self-supervised representation
Metrics: R^2 Kendall tau test MSE
Baselines: DeepSets DMC MLP MLP-Aug NFN-PT NFN-HNP NFN-NP NFT INR2Vec DWSNets MLP + Perm. aug MLP + Alignment
Builds on: Deep Weight Space Networks (DWSNets) Neural Functional Networks (NFN) Neural Functional Transformers (NFT) Deep Meta Classifier (DMC) DeepSets
Closest papers by meaning
- Unified Graph Neural Networks Pre-training for Multi-domain Graphs (AAAI 2025 · similarity 0.89)
- Scale Equivariant Graph Metanetworks (NeurIPS 2024 · similarity 0.88)
- Graph Neural Networks Are More Than Filters: Revisiting and Benchmarking from A Spectral Perspective (ICLR 2025 · similarity 0.88)
- Generalizing CNNs to graphs with learnable neighborhood quantization (NeurIPS 2024 · similarity 0.87)
- A Manifold Perspective on the Statistical Generalization of Graph Neural Networks (ICML 2025 · similarity 0.87)
- How Graph Neural Networks Learn: Lessons from Training Dynamics (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.