AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Commutative Scaling of Width and Depth in Deep Neural Networks

Soufiane Hayou

JMLR 2024 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase width n from 8 to 256 to 16384ResNet with uniform scaling alpha=L^-1/2 · Neural covariance at depth L=2L2 error to theory: 2.34e-01 -> 1.32e-02 -> 1.04e-03Improved
increase depth L from 2 to 256 at width 2^14ResNet with uniform scaling alpha=L^-1/2 · Neural covariance, width 16384L2 error to theory: 1.04e-03 -> 1.62e-04Improved
increase depth L from 2 to 256 at width 8ResNet with uniform scaling alpha=L^-1/2 · Neural covariance, width 8L2 error to theory: 2.34e-01 -> 6.08e-01Hurt
use shaped ReLU MLP instead of scaled ResNetProportional width-depth limit L=n · Neural covariance distributionconcentration around sequential limitHurt
use Shaped ResNet with main branch scaled by beta=1/2 instead of alpha=L^-1/2Residual network · Neural covariance distribution, L=nconcentration around sequential limitHurt
improved depth rate analysis versus prior rate L^-1/2Infinite-width covariance for alpha=L^-1/2 · Delta_L vs depth L in 2^3..2^14convergence rate in L: L^-1/2 -> L^-1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy