Research map / Architectures and efficiency / RNNs and normalization

Most ReLU Networks Admit Identifiable Parameters

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows via weighted polyhedral complexes that for every ReLU architecture with input and hidden widths at least two there is an open set of identifiable parameters, giving functional dimension = #params − #hidden neurons, plus minimal non-identifiable open sets and a generic depth hierarchy.

Problem

Which deep ReLU architectures attain the expected functional dimension, i.e., when does a function determine its parameters up to scaling and permutation (identifiability), and can non-identifiability persist for minimal realizations?

Approach

Introduces a weighted polyhedral complex framework separating the canonical complex from the visible breakpoint complex, uses bending ridges to build a dependency graph ordering neurons by layer, and combines this with a reverse-engineering argument and rank conditions to remove the last-layer sign ambiguity, yielding identifiability among all parameters on an open set for general widths ≥2.

Key results

• For every architecture with input and hidden widths at least two, there exists an open set of parameters identifiable among all parameters, so functional dimension equals number of parameters minus number of hidden neurons.
• There exists an open set of minimal parameters with positive-dimensional nontrivial fibers, showing minimality does not imply identifiability.
• A generic depth hierarchy: for an open set of parameters the realized function cannot be represented generically by any shallower network, regardless of width.

Surprising findings

• Minimal realizations can still carry positive-dimensional hidden symmetries, so minimality does not imply identifiability.
• Prior reverse-engineering uniqueness results hold only within restricted TPIC/LRA classes, not among all generic parameters.

Limitations

• Result requires input and hidden layers of width at least two; the excluded cases are not fully settled.
• Existence results on open sets are non-constructive in character; identifiability is shown for some open set rather than characterized explicitly.

Experimental setup

Builds on: reverse-engineering of ReLU networks via bent hyperplanes tropical weights of CPWL functions transparent ReLU layers

Closest papers by meaning

• The Symmetries of Three-Layer ReLU Networks (NeurIPS 2026 · similarity 0.90)
• Functional vs. parametric equivalence of ReLU networks (ICLR 2020 · similarity 0.90)
• Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.89)
• On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth (ICLR 2025 · similarity 0.87)
• Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.87)
• On the Local Complexity of Linear Regions in Deep ReLU Networks (ICML 2025 · similarity 0.86)