Research map / Architectures and efficiency / RNNs and normalization
Emergent Equivariance in Deep Ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that infinitely wide deep ensembles trained with full data augmentation are exactly equivariant at all training times and inputs (off-manifold too), via NTK theory, and verifies this on Ising, FashionMNIST and histological data.
Problem
Whether deep ensembles of neural networks, trained with data augmentation, become equivariant to data symmetries even though individual members are not equivariant, including off the data manifold, at initialization, and at any training time.
Approach
Using the infinite-width NTK/NNGP Gaussian-process description of ensemble mean predictions, the kernels are shown to transform covariantly under the input group action; combined with data augmentation (which turns group actions into permutations of the training set), the ensemble mean is exactly equivariant for any architecture in the infinite width limit, and approximately equivariant at finite
Key results
- On the Ising model, ensemble-mean predictions deviate from invariance by about 0.8% out of distribution for large ensembles and widths, versus 82% for individual members.
- On FashionMNIST OOD (MNIST), ensemble of size 100 with C16 augmentation reaches 15.24±0.69 orbit-same-prediction, competitive with E2CNN (15.08±0.34) and canonicalization (12.41±0.85); data augmentation alone reaches 12.77±0.71.
- For SO(2) rotations, C16-augmented ensembles leave over 90% of orbit elements with the same prediction as the unrotated input.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| average predictions across ensemble members instead of using data augmentation alone | FashionMNIST CNNs · OOD MNIST validation set, C4/C8/C16 | orbit same prediction: 3.41±0.18 -> 3.85±0.12; 6.73±0.24 -> 7.72±0.34; 12.77±0.71 -> 15.24±0.69 | Helped |
| average predictions across ensemble members rather than use an individual member | Ising deep ensembles · Ising OOD set | relative standard deviation across orbits: 82% -> 0.8% | Helped |
| increase the number of ensemble members | FashionMNIST deep ensembles · OOD data | orbit same prediction | Helped |
| increase the augmentation group order | FashionMNIST deep ensembles · SO(2) rotations | fraction of orbit elements with the same prediction: over 90 percent | Helped |
| train with full data augmentation | infinite-width deep ensembles · all inputs and training times | equivariance | Helped |
Surprising findings
- Individual ensemble members are highly non-equivariant (82% RSD out of distribution on Ising) while their average is nearly invariant, so equivariance emerges from averaging alone.
- Manifestly equivariant E2CNN and canonicalized models are themselves slightly broken for group orders above 4 due to interpolation artifacts.
Limitations
- Exact equivariance holds only in the infinite-width NTK limit; finite-width corrections are not quantified.
- Continuous groups and random (non-subgroup) augmentation only give approximate equivariance, with the random-augmentation effect not controlled by the theory.
- Theory covers MSE training and specific layer types (conv, FC, flattening, graph aggregation); extension to attention, pooling, dropout is claimed but not shown.
Experimental setup
Datasets: FashionMNIST MNIST CIFAR10 (grey-scaled) NCT-CRC-HE-100K 2D Ising model (5x5 lattice)
Benchmarks: FashionMNIST OOD (MNIST, grey-scaled CIFAR10, Gaussian noise) NCT-CRC-HE-100K OOD Ising energy invariance
Metrics: relative standard deviation across orbit orbit same prediction (OSP)
Baselines: E2CNN canonicalization network data augmentation without ensembling
Builds on: neural tangent kernel NNGP Equivariant CNNs (E2CNN) neural-tangents
Closest papers by meaning
- Equivariant Neural Tangent Kernels (ICML 2025 · similarity 0.88)
- Approximately Equivariant Neural Processes (NeurIPS 2024 · similarity 0.87)
- A Probabilistic Approach to Learning the Degree of Equivariance in Steerable CNNs (ICML 2024 · similarity 0.86)
- Flow Equivariant Recurrent Neural Networks (NeurIPS 2025 · similarity 0.86)
- Learning (Approximately) Equivariant Networks via Constrained Optimization (NeurIPS 2025 · similarity 0.85)
- On Universality Classes of Equivariant Networks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.