AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Topology of Deep Neural Networks

Gregory Naitzat, Andrey Zhitnikov, Lek-Heng Lim

JMLR 2020 · Architectures and efficiency › RNNs and normalization · approach: Symmetry and equivariance in loss landscapes · 93 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use ReLU instead of tanh activation (ablation)deep fully connected classifier · D-I, D-II, D-III simulated manifoldsreduction of Betti numbers across layersImproved
use leaky ReLU instead of ReLU activation (ablation)deep fully connected classifier · simulated manifoldsspeed of Betti number reductionHurt
use tanh with lower floating point precision (larger unit roundoff)tanh network · topology changetopology-changing ability: delta 2^-54 -> 2^-9Improved
add depth to constant-width network (ablation)fifteen-neuron-wide networks · simulated manifoldsfraction of networks trained to high accuracyImproved
shift topological work to final layers when depth is reduced (ablation)shallower networks · simulated manifoldslayerwise Betti reductionMixed
add a three-neuron bottleneck layer to ten-layer networks (ablation)width-15 network · D-Itopology changeImproved
use narrow networks (6 neurons) instead of wide (50 neurons) (ablation)ten-layer network · D-Ispeed of topology changeImproved
increase width from 15 to 50 neurons (ablation)ten-layer network · D-II and D-IIItopology changeNo clear effect
reduce width below fifteen neurons (ablation)ten-layer network · simulated datapercentage of successfully trained networksHurt
use tanh instead of ReLU or leaky ReLU (ablation)ten-layer network · D-Ivariance of beta_0 decayHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy