AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Which Neural Net Architectures Give Rise to Exploding and Vanishing Gradients?

Boris Hanin

NeurIPS 2018 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 37 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Increase the sum of reciprocals of hidden-layer widths, βRandom fully connected ReLU networks at initialization · Fully connected ReLU networksVariance of squared Jacobian entries: exponential in \betaHurt
Increase hidden-layer widths so that \sum_{j=1}^{\infty}\frac{1}{n_j}<\inftyRandom fully connected ReLU networks at initialization · Deep fully connected ReLU networksGradient stability: \sum_{j=1}^{\infty}\frac{1}{n_j}<\inftyImproved
Make hidden-layer widths equal while holding depth and total hidden-neuron budget fixedFully connected ReLU networks · Deep fully connected ReLU networksGradient stabilityImproved
Vary the weight distributions while keeping weights independent, zero-centered, and correctly variance-scaledRandom fully connected ReLU networks · Fully connected ReLU networksOccurrence of the EVGPNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy