Research map / Architectures and efficiency / RNNs and normalization

The Symmetries of Three-Layer ReLU Networks

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Characterizes the generic parameter fibers of three-layer bottleneck ReLU networks via translation and sign-flipping symmetries, yielding a polynomial-time functional-equivalence test and showing some symmetries give locally conserved quantities under gradient flow.

Problem

Parameter redundancy (fibers) in deep ReLU networks is largely uncharacterized beyond single-hidden-layer networks; it is unknown how symmetries from layer composition shape fibers in networks with more than one hidden layer.

Approach

Shows that a first-layer hyperplane hidden by the second layer can be translated with compensating second-layer bias shifts, and for single-neuron second layers can be sign-flipped via a finite group; for generic bottleneck three-layer networks these symmetries are complete, giving an explicit fiber description and a polynomial-time equivalence test.

Key results

• For generic bottleneck three-layer ReLU networks, non-identifiability holds iff the second-layer weight/bias matrix has an all-positive or all-negative row, or a row positive in exactly one first-layer column and negative elsewhere.
• A polynomial-time algorithm decides functional equivalence of two generic bottleneck parameters.
• The volume of the identifiable set is bounded: 1 - n2 2^{-n1} - n1 2^{-n2 n1} <= ratio <= (1 - 2^{-(n1+1)})^{n2}.

Surprising findings

• Symmetries of composed layers are not localizable: pairwise compositions can be identifiable while the full three-layer composition is not.
• The hiding-based symmetries from the paper do not induce locally conserved quantities under gradient flow, unlike the linear-neuron symmetries.

Limitations

• Results hold only for generic parameters and bottleneck architectures with n1 <= d.
• Completeness and explicit descriptions beyond the three-layer bottleneck regime are left open.
• Number of connected components of fibers is not characterized.

Experimental setup

Builds on: hyperplane representation of one-hidden-layer ReLU networks balanced polyhedral complexes from tropical geometry

Closest papers by meaning

• Hidden Symmetries of ReLU Networks (ICML 2023 · similarity 0.90)
• Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.90)
• Symmetries, Flat Minima, and the Conserved Quantities of Gradient Flow (ICLR 2023 · similarity 0.89)
• Topological obstruction to the training of shallow ReLU neural networks (NeurIPS 2024 · similarity 0.87)
• Transformative or Conservative? Conservation laws for ResNets and Transformers (ICML 2025 · similarity 0.86)
• A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizations (ICML 2025 · similarity 0.86)