Research map / Architectures and efficiency / RNNs and normalization
Equivariant Polynomial Functional Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAGEP-NFN, a parameter-sharing neural functional network whose equivariant and invariant layers are polynomials built from stable polynomial terms, equivariant to permutation and positive scaling of MLP/CNN weights.
Problem
Designing permutation- and scaling-equivariant neural functional networks that keep low memory and runtime (as parameter-sharing linear layers do) while being more expressive than linear equivariant layers, which only update each layer's weights from the same layer.
Approach
Replace linear parameter-sharing layers with nonlinear polynomial layers in the input weights built from "stable polynomial terms" (products of weight matrices across layers, with biases and learnable connection matrices) that transform equivariantly under the monomial matrix group; these capture inter-layer relations while staying cheap.
Key results
- INR classification on MNIST: 77.55% vs 69.82% for the next best (NP), a 7.73-point gain.
- CNN generalization prediction on ReLU subset: Kendall's tau 0.933 vs 0.926 for HNP on original data; 0.940 on Tanh subset vs 0.939 for Monomial-NFN.
- Style editing: matches NP and Monomial-NFN on CIFAR-10 contrast (MSE 0.020) and ties NP on MNIST dilation (0.068).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add polynomial Inter-Layer terms to Non Inter-Layer terms (ablation) | MAGEP-NFNs · Small CNN Zoo ReLU, generalization prediction | Kendall's τ: 0.929 -> 0.933 | Helped |
| Use only Inter-Layer terms instead of the full combination (ablation) | MAGEP-NFNs · Small CNN Zoo ReLU, generalization prediction | Kendall's τ: 0.933 -> 0.932 | Hurt |
| Add [W] to Non Inter-Layer terms (ablation) | MAGEP-NFNs · Small CNN Zoo ReLU, generalization prediction | Kendall's τ: 0.929 -> 0.930 | Helped |
| Add [WW] to Non Inter-Layer terms (ablation) | MAGEP-NFNs · Small CNN Zoo ReLU, generalization prediction | Kendall's τ: 0.929 -> 0.930 | Helped |
| Add [Wb] to Non Inter-Layer terms (ablation) | MAGEP-NFNs · Small CNN Zoo ReLU, generalization prediction | Kendall's τ: 0.929 -> 0.931 | Helped |
| Add [bW] to Non Inter-Layer terms (ablation) | MAGEP-NFNs · Small CNN Zoo ReLU, generalization prediction | Kendall's τ: 0.929 -> 0.931 | Helped |
| Use MAGEP-NFNs instead of HNP | Small CNN Zoo ReLU, no augment · Small CNN Zoo ReLU, no augment | Kendall's τ: 0.926 \pm 0.003 -> 0.933 \pm 0.001 | Helped |
| Use MAGEP-NFNs instead of Monomial-NFN | Small CNN Zoo ReLU with scale augmentation · Small CNN Zoo ReLU, U[1, 10] | Kendall's τ: 0.920 \pm 0.001 -> 0.933 \pm 0.001 | Helped |
| Use MAGEP-NFNs instead of Monomial-NFN | Small CNN Zoo Tanh · Small CNN Zoo Tanh | Kendall's τ: 0.939 \pm 0.0004 -> 0.940 \pm 0.001 | No clear effect |
| Use MAGEP-NFNs instead of NP | MNIST INR classification · MNIST INR classification | accuracy: 69.82 \pm 0.42 -> 77.55 \pm 0.68 | Helped |
| Use MAGEP-NFNs instead of Monomial-NFN | CIFAR-10 INR classification · CIFAR-10 INR classification | accuracy: 34.23 \pm 0.33 -> 37.18 \pm 0.30 | Helped |
| Use MAGEP-NFNs instead of Monomial-NFN | FashionMNIST INR classification · FashionMNIST INR classification | accuracy: 61.15 \pm 0.55 -> 62.83 \pm 0.57 | Helped |
Surprising findings
- NP remains a strong candidate on weight editing, and MAGEP-NFN only performs on par with it rather than clearly better.
Limitations
- Equivariant polynomial layers are derived only for a specific architecture (MLPs/CNNs with ReLU), not generalized to other operators like normalization or pooling.
- Gains over baselines are often small or absent (e.g., weight editing tie with NP).
Experimental setup
Datasets: Small CNN Zoo INR weight datasets for MNIST, FashionMNIST, CIFAR-10 SIREN weights for CIFAR-10 and MNIST
Benchmarks: Small CNN Zoo INR classification on MNIST, FashionMNIST, CIFAR-10 weight space style editing (contrast CIFAR-10, dilation MNIST)
Metrics: accuracy Kendall's tau MSE
Baselines: MLP NP HNP Monomial-NFN STATNet
Builds on: Monomial-NFN neural functional networks (NFN) parameter sharing for equivariant layers
Closest papers by meaning
- Equivariant Neural Functional Networks for Transformers (ICLR 2025 · similarity 0.89)
- Monomial Matrix Group Equivariant Neural Functional Networks (NeurIPS 2024 · similarity 0.88)
- A Characterization Theorem for Equivariant Networks with Point-wise Activations (ICLR 2024 · similarity 0.87)
- Scale Equivariant Graph Metanetworks (NeurIPS 2024 · similarity 0.87)
- Are High-Degree Representations Really Unnecessary in Equivariant Graph Neural Networks? (NeurIPS 2024 · similarity 0.86)
- On Universality Classes of Equivariant Networks (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.