Research map / Learning theory and algorithms / Prediction and decision losses
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use orthogonal rather than Gaussian weights | Deep linear networks · Deep linear network | Jacobian singular-value concentration: all its singular values are 1 | Helped |
| Use Gaussian rather than orthogonal weights | Deep linear networks · Deep linear network | variance of eigenvalue distribution: σ^2_{JJ^T} = L | Hurt |
| Use orthogonal weights in a ReLU network | Critical ReLU networks · Critical ReLU network | variance of eigenvalue distribution: σ_{JJ^T}^2 = \frac{1-p(q^*)}{p(q^*)} L | Hurt |
| Use orthogonal instead of Gaussian weights | Critical hard-tanh networks · Critical hard-tanh network | Jacobian conditioning: λ_max can remain O(1) in the former case but grows linearly in the latter | Helped |
| Reduce q* along the critical line, increasing the fraction of neurons in the linear regime | Orthogonal hard-tanh networks · Critical hard-tanh network | Jacobian conditioning: p(q*) ≈ 1-\frac{1}{L} | Helped |
| Replace orthogonal initialization with Gaussian initialization | Tanh networks with small σ_w^2 · CIFAR-10 | training speed | Hurt |
| Use orthogonal tanh with small σ_w^2 rather than ReLU | Deep networks · CIFAR-10 | training speed: orders of magnitude faster | Helped |
| Use a small nonzero q* instead of q*=0 (ablation) | Orthogonal tanh networks · CIFAR-10 | training speed | Helped |
| Use a small nonzero q* instead of q*=0 (ablation) | Orthogonal tanh networks · CIFAR-10 | generalization accuracy | Helped |
| Use different optimizers while retaining the initialization (ablation) | Critical networks · CIFAR-10 | training speed advantage of isometric initialization | No clear effect |
| Increase depth in orthogonal tanh networks | Orthogonal tanh networks · CIFAR-10 | training time: O(\sqrt{L}) | Hurt |
Closest papers by meaning
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.88)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.87)
- On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.87)
- One-Time Soft Alignment Enables Resilient Learning without Weight Transport (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.