Research map / Architectures and efficiency / CNN design
Monomial Matrix Group Equivariant Neural Functional Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Monomial-NFN, neural functional networks equivariant to monomial matrix groups (permutations plus ReLU scaling or sin/tanh sign flipping) for FCNN/CNN weight spaces, with far fewer parameters than permutation-equivariant NFNs.
Problem
Existing neural functional networks (NFNs) only respect neuron permutation symmetry and ignore weight scaling symmetries of ReLU networks and sign-flipping symmetries of sin/tanh networks, so functionally equivalent weights can yield different outputs.
Approach
Extend the symmetry group of weight spaces from permutation matrices to monomial matrices (permutation times diagonal scaling/sign), derive the maximal such groups preserved by ReLU (positive scaling) and sin/tanh (sign flipping), and build equivariant and invariant linear layers for them; the resulting parameter count is linear rather than quadratic in depth.
Key results
- On the Tanh subset of Small CNN Zoo, Monomial-NFN reaches Kendall's tau 0.939 on original data vs 0.933 for HNP, and 0.943 vs 0.935 on augmented data, using up to 50% of HNP's parameters.
- On INR classification, Monomial-NFN reaches 34.23% on CIFAR-10 (vs NP 33.74%) and 61.15% on FashionMNIST (vs NP 58.21%), but 68.43% on MNIST vs NP 69.82%.
- Equivariant layer parameter count is O(cc'(L+n_0+n_L)) versus O(cc'(L+n_0+n_L)^2) for permutation-equivariant baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Monomial-NFN instead of HNP | CNN generalization prediction on Small CNN Zoo Tanh · Small CNN Zoo Tanh (original) | Kendall's τ: 0.933 \pm 0.002 -> 0.939 \pm 0.001 | Helped |
| Train with augmented data | Monomial-NFN for CNN generalization prediction · Small CNN Zoo Tanh | Kendall's τ: 0.939 \pm 0.001 -> 0.943 \pm 0.001 | Helped |
| Use Monomial-NFN instead of NP | INR image classification · FashionMNIST | test accuracy (%): 58.21 \pm 0.31 -> 61.15 \pm 0.55 | Helped |
| Use Monomial-NFN instead of NP | INR image classification · CIFAR-10 | test accuracy (%): 33.74 \pm 0.26 -> 34.23 \pm 0.33 | Helped |
| Use Monomial-NFN instead of NP | INR image classification · MNIST-10 | test accuracy (%): 69.82 \pm 0.42 -> 68.43 \pm 0.51 | Hurt |
| Use Monomial-NFN instead of NP | Weight-space style editing · Contrast editing (CIFAR-10) | test mean squared error: 0.020 \pm 0.002 -> 0.020 \pm 0.001 | No clear effect |
| Use Monomial-NFN instead of NP | Weight-space style editing · Dilate editing (MNIST) | test mean squared error: 0.068 \pm 0.002 -> 0.069 \pm 0.002 | Hurt |
| Apply larger weight-scaling and permutation augmentations | CNN generalization prediction baselines HNP and STATNN · Small CNN Zoo ReLU | Kendall's τ | Hurt |
| Apply larger weight-scaling and permutation augmentations | Monomial-NFN · Small CNN Zoo ReLU | Kendall's τ | No clear effect |
Surprising findings
- On the MNIST INR classification task, Monomial-NFN is outperformed by NP (68.43% vs 69.82%).
Limitations
- Linear equivariant layers may be limited in expressivity because the large group leaves each weight updated only from its own previous value.
- Maximality of the symmetry group G for ReLU networks is not established.
- Fourier features and positional embeddings are incompatible with the scaling-equivariant design.
Experimental setup
Datasets: Small CNN Zoo INR dataset of pretrained INRs for CIFAR-10, MNIST, FashionMNIST SIREN weights from HNP paper
Benchmarks: Small CNN Zoo (Tanh and ReLU subsets) INR classification on CIFAR-10, MNIST, FashionMNIST SIREN weight-space editing (contrast CIFAR-10, dilate MNIST)
Metrics: Kendall's tau classification accuracy test MSE
Baselines: STATNN NP HNP MLP
Builds on: Neural functional networks (permutation equivariant) NFN Hierarchical Neural Program (HNP) STATNN
Closest papers by meaning
- Learning symmetries via weight-sharing with doubly stochastic tensors (NeurIPS 2024 · similarity 0.88)
- On Universality Classes of Equivariant Networks (NeurIPS 2025 · similarity 0.88)
- Equivariant Polynomial Functional Networks (ICML 2025 · similarity 0.88)
- Compact Matrix Quantum Group Equivariant Neural Networks (ICML 2025 · similarity 0.88)
- Lie Group Decompositions for Equivariant Neural Networks (ICLR 2024 · similarity 0.87)
- Scale Equivariant Graph Metanetworks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.