Research map / Architectures and efficiency / RNNs and normalization
Separation Power of Equivariant Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes which inputs are indistinguishable by equivariant neural networks with regular G-convolutions (finite groups, permutation representations, pointwise activations), via a recursive twin-network zero-locus formula, and derives how activation, depth, width, and representation type affect separation power.
Problem
Understanding the separation power (ability to distinguish distinct inputs) of equivariant neural networks beyond graph settings, which is a necessary condition for fine-grained universality and expressivity analysis.
Approach
Reformulates the identification relation of a neural space as a zero-locus problem on twin networks, yielding a recursive formula over depth; uses it to show all non-polynomial activations are equivalent, depth helps up to a stabilization threshold, invariant hidden features do not matter, and minimal permutation representations form a subgroup-based hierarchy.
Key results
- Any continuous non-polynomial activation (ReLU, sigmoid, tanh) yields the same separation power, which is maximal for a fixed architecture.
- Depth increases separation power only up to a repetition threshold after which the identification relation stabilizes.
- Adding invariant hidden features (block multiplicity) does not change separation power; any k-IGN with arbitrary hidden feature widths matches k-WL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a continuous non-polynomial activation instead of another continuous activation | Equivariant neural networks with a fixed architecture · General equivariant inputs | separation power | Helped |
| Increase the number of repeated layers of the same type | Equivariant neural networks with an identity map in the repeated layer space · General equivariant inputs | separation power | Mixed |
| Increase repeated regular-representation layers beyond one repetition | Networks with regular hidden representations · General equivariant inputs | separation power | No clear effect |
| Increase the multiplicity of hidden features | Equivariant neural networks · General equivariant inputs | separation power | No clear effect |
| Add invariant features to hidden representations | k-IGN · Graph inputs | separation power | No clear effect |
| Use a larger filter support | Shallow circular CNNs · Circular CNN inputs, n > 2 | separation power | Helped |
| Replace a hidden representation with a sum of two representation components | Equivariant neural networks · General equivariant inputs | separation power | Helped |
| Use a larger transitive representation corresponding to a smaller subgroup | Equivariant neural networks · General equivariant inputs | separation power | Helped |
| Use a regular hidden representation instead of a trivial representation | Equivariant neural networks · General equivariant inputs | separation power | Helped |
| Use any positive hidden feature dimensions instead of requiring large hidden feature dimensions | k-IGN · Graph inputs | separation power | No clear effect |
Surprising findings
- Hidden feature width, i.e. multiplicity of invariant blocks, has no effect on separation power, strengthening prior results that required large widths for IGNs to match k-WL.
Limitations
- Only applies to permutation representations and finite groups; does not cover Clebsch-Gordan or polynomial equivariant models.
- Assumes layers with bias; bias-free models such as some GNNs only receive bounds.
- Relies on non-polynomial activations; some polynomial activations may also be maximal but are not characterized.
Experimental setup
Baselines: k-WL k-IGN
Builds on: Weisfeiler-Leman test k-IGN regular G-convolutions
Closest papers by meaning
- On Universality Classes of Equivariant Networks (NeurIPS 2025 · similarity 0.91)
- On Universality of Deep Equivariant Networks (ICLR 2026 · similarity 0.89)
- A Characterization Theorem for Equivariant Networks with Point-wise Activations (ICLR 2024 · similarity 0.85)
- A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models (NeurIPS 2025 · similarity 0.85)
- A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs (ICML 2024 · similarity 0.85)
- On the Hölder Stability of Multiset and Graph Neural Networks (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.