Research map / Architectures and efficiency / RNNs and normalization
Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Partially linearize the network after training rather than during early training (ablation) | ResNet56 Short · CIFAR-10 and harder image-classification datasets | test accuracy | Helped |
| Partially linearize later rather than earlier in training (ablation) | Transformer · Multi30k German-to-English | test perplexity | Helped |
| Train with the same nonlinear-unit mask from scratch instead of extracting it after training (ablation) | ResNet56 · CIFAR-100 | test accuracy | Hurt |
| Train with exact or layer-wise transferred nonlinear-unit masks from scratch (ablation) | ResNet56 · CIFAR-10 | test accuracy | No clear effect |
| Globally permute the nonlinear-unit mask (ablation) | ResNet56 NoShort · CIFAR-10 | test accuracy | Hurt |
| Increase the nonlinear-unit regularization strength (ablation) | ResNet56 Short · CIFAR-10, CINIC-10, and CIFAR-100 | test accuracy | Hurt |
| Increase task difficulty (ablation) | Partially linearized ResNet56 Short · CIFAR-10, CINIC-10, and CIFAR-100 | NAPL | Mixed |
| Use wider initial networks (ablation) | ResNet56 Short · CIFAR-10 | test accuracy | Helped |
| Vary initial network depth (ablation) | ResNet · CIFAR-10 | NAPL | No clear effect |
| Use residual connections rather than a plain network (ablation) | ResNet56 · CIFAR-10 and CIFAR-100 | distribution of remaining nonlinear units | Mixed |
| Increase partial linearization (ablation) | ResNet50 feature visualizations · ImageNet | feature-image sharpness | Helped |
Closest papers by meaning
- From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport (CVPR 2025 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.87)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.86)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.86)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.