Research map / Architectures and efficiency / RNNs and normalization
Low-dimensional topology of deep neural networks
Junyu Ren, Lek-heng Lim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ReLU with nonmonotonic GELU (ablation) | width-3 feedforward, depth 8 · Hopf link classification | held-out accuracy (%): 63.1 -> 91.1 | Improved |
| replace ReLU with nonmonotonic GELU (ablation) | width-3 feedforward, depth 3 · Hopf link classification | held-out accuracy (%): 84.3 -> 89.3 | Improved |
| replace ReLU with nonmonotonic GELU (ablation) | width-3 feedforward, depth 20 · Hopf link classification | held-out accuracy (%): 50.3 -> 52.8 | No clear effect |
| replace ReLU with GELU: best-seed ceiling (ablation) | width-3 feedforward, depth 5 · Hopf link classification | max accuracy over seeds (%): 92.5 -> 100.0 | Improved |
| add skip connections (ReLU ResNet) (ablation) | width-3 ReLU network, depth 5 · Hopf link classification | held-out accuracy (%): 74.3 -> 98.5 | Improved |
| add skip connections (ReLU ResNet) (ablation) | width-3 ReLU network, depth 3 · Hopf link classification | held-out accuracy (%): 83.9 -> 97.5 | Improved |
| add skip connections (ablation) | width-5 ReLU, linked S2 pair k=1 · Linked S2 x S2 in R^5 | best test accuracy (%): 98.6 -> 100.0 | Improved |
| add skip connections (ablation) | width-5 ReLU, k=10 linked copies · Linked S2 x S2 in R^5 | best test accuracy (%): 87.9 -> 85.4 | Hurt |
| replace ReLU with GELU (ablation) | width-5 depth-5 network, k=50 · Linked S2 x S2 in R^5 | best test accuracy (%): 80.2 -> 84.8 | Improved |
| replace ReLU with Swish (ablation) | width-5 depth-5 network, k=10 · Linked S2 x S2 in R^5 | best test accuracy (%): 87.9 -> 91.6 | Improved |
| use nonmonotonic activation (GELU) instead of ReLU (ablation) | L8 no-skip CNN · CIFAR-10 deer-dog (linked pair) | accuracy (%): 89.6 -> 90.8 | Improved |
| use nonmonotonic activation (GELU) instead of ReLU (ablation) | L8 no-skip CNN · CIFAR-10 frog-ship (unlinked control) | accuracy (%): 97.8 -> 97.9 | No clear effect |
Closest papers by meaning
- Topological Blindspots: Understanding and Extending Topological Deep Learning Through the Lens of Expressivity (ICLR 2025 · similarity 0.87)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.86)
- Weisfeiler and Lehman Go Paths: Learning Topological Features via Path Complexes (AAAI 2024 · similarity 0.86)
- Input Space Mode Connectivity in Deep Neural Networks (ICLR 2025 · similarity 0.86)
- Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture (ICLR 2025 · similarity 0.86)
- A Topological Perspective on Demystifying GNN-Based Link Prediction Performance (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
