Research map / Point clouds / Point cloud classification
Fixed points of nonnegative neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses nonlinear Perron-Frobenius theory to show nonnegative neural networks are monotone and weakly scalable, deriving fixed-point existence, uniqueness, and fixed-point-set shape conditions, illustrated on nonnegative autoencoders on MNIST.
Problem
Existence, uniqueness, and shape of fixed point sets of nonnegative neural networks (nonnegative inputs mapped to nonnegative outputs, matching input/output dimensions), motivated by autoencoders and deep equilibrium models.
Approach
Recognizes nonnegative neural networks with nonnegative weights and biases as monotonic and (weakly) scalable mappings in nonlinear Perron-Frobenius theory, giving fixed-point existence conditions (spectral radius of the asymptotic linear mapping below 1) weaker than convex-analysis (operator-norm) conditions, and showing the fixed point set is an interval that degenerates to a point under scalabi
Key results
- Two-layer nonnegative sigmoid autoencoder has spectral radius 0 and thus a unique positive fixed point, per the theory.
- ReLU autoencoder trained without constraint had spectral radius 1.01 > 1; spectral normalization brought it to 0.93 on MNIST, yielding a nondegenerate fixed point set.
- Tanh NN autoencoder reached test loss 0.0063 vs 0.2308 for Sigmoid NN on MNIST (averaged over 10 runs).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sigmoid activations with tanh activations | NN autoencoder · Autoencoder test set | test loss: 0.2308± 2.3\cdot 10^{-6} -> 0.0063\pm3.9\cdot 10^{-6} | Helped |
| use positive weights instead of nonnegative weights | Tanh NN autoencoder · Autoencoder test set | test loss: 0.0063\pm3.9\cdot 10^{-6} -> 0.0246\pm 5.6\cdot 10^{-5} | Hurt |
| replace tanh activations with ReLU and apply spectral normalization | NN autoencoder · Autoencoder test set | test loss: 0.0063\pm3.9\cdot 10^{-6} -> 0.0061\pm 1.1\cdot 10^{-5} | Helped |
| use positive weights instead of nonnegative weights | ReLU spectral NN autoencoder · Autoencoder test set | test loss: 0.0061\pm 1.1\cdot 10^{-5} -> 0.0059\pm 1.3\cdot 10^{-5} | Helped |
| replace tanh with tanh + Swish activations | NN autoencoder · Autoencoder test set | test loss: 0.0063\pm3.9\cdot 10^{-6} -> 0.0041\pm 2.5\cdot 10^{-6} | Helped |
| use ReLU + sigmoid activations and allow real-valued biases | Tanh NN autoencoder · Autoencoder test set | test loss: 0.0063\pm3.9\cdot 10^{-6} -> 0.0031\pm 4.7\cdot 10^{-5} | Helped |
| allow real-valued weights and biases | ReLU + Sigmoid NR autoencoder · Autoencoder test set | test loss: 0.0031\pm 4.7\cdot 10^{-5} -> 0.0042\pm 7.7\cdot 10^{-5} | Hurt |
| apply spectral normalization to the ReLU autoencoder | ReLU NN autoencoder · MNIST | nonlinear spectral radius: 1.01 -> 0.93 | Hurt |
| apply spectral normalization to the ReLU autoencoder | ReLU NN autoencoder · ZERO | nonlinear spectral radius: 1.01 -> 0.91 | Hurt |
| apply spectral normalization to the ReLU autoencoder | ReLU NN autoencoder · MNIST and ZERO | test loss: 0.0061 -> 0.0059 | Helped |
Surprising findings
- The sigmoid nonnegative autoencoder performed much worse (test loss 0.2308) than the other configurations.
- For tanh NN, the shape of the fixed point set could not be proven (Lemma primitivity condition not satisfied), despite empirical evidence it is an interval.
Limitations
- Shape of fixed point set for some configurations (tanh NN, ReLU spectral NN) only supported empirically, not proven.
- Experiments limited to small two-layer autoencoders on MNIST.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: test loss (mean squared error) spectral radius of asymptotic mapping products of spectral norms of weight matrices
Baselines: convex-analysis operator-norm bounds on fixed point existence
Builds on: nonlinear Perron-Frobenius theory Thompson metric spectral normalization deep equilibrium models
Closest papers by meaning
- [Re] Nondeterminism and Instability in Neural Network Optimization (NeurIPS 2022 · similarity 0.83)
- Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.80)
- Exploring The Loss Landscape Of Regularized Neural Networks Via Convex Duality (ICLR 2025 · similarity 0.79)
- REGLO: Provable Neural Network Repair for Global Robustness Properties (AAAI 2024 · similarity 0.79)
- Inexact Newton-type Methods for Optimisation with Nonnegativity Constraints (ICML 2024 · similarity 0.79)
- Koopman-based generalization bound: New aspect for full-rank weights (ICLR 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.