Research map / Architectures and efficiency / RNNs and normalization
Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes using the variance of per-channel weight norms during training as an indicator of whether a layer is wide enough (IS vs DS patterns), and uses it to redistribute width across VGG/ResNet layers for fewer parameters and better accuracy.
Problem
Layer widths in deep networks are mostly chosen empirically; there is no practical training-time indicator of whether a layer has sufficient width or how to allocate width across layers.
Approach
Track the variance across channels of the product of incoming and outgoing weight norms during training. Wide layers show an increase-to-saturate (IS) pattern, while narrow layers show a decrease-to-saturate (DS) pattern. Middle layers of conventional CNNs show DS, so widening them and narrowing the others (width streamlining) reduces parameters while keeping FLOPs similar and improving accuracy.
Key results
- Width-streamlined ResNet18 on CIFAR-10 reaches 93.60 ± 0.13% vs 92.99 ± 0.23% for the original, with 55.99% of the parameters and 108.30% of the FLOPs.
- Width-streamlined VGG16 on CIFAR-100 reaches 68.63 ± 0.31% vs 66.17 ± 0.71% for the original, with 61.55% of the parameters.
- On Tiny-ImageNet, streamlined VGG16 improves validation accuracy from 49.78% to 54.40%, and ResNet18 from 54.46% to 56.56%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adjust layer widths using width-wise streamlining | VGG16 · CIFAR-10 | Top-1 accuracy: 91.02 ± 0.35 -> 93.41 ± 0.15 | Helped |
| Adjust layer widths using width-wise streamlining | VGG16 · CIFAR-100 | Top-1 accuracy: 66.17 ± 0.71 -> 68.63 ± 0.31 | Helped |
| Adjust layer widths using width-wise streamlining | VGG19 · CIFAR-10 | Top-1 accuracy: 91.84 ± 0.46 -> 92.38 ± 0.09 | Helped |
| Adjust layer widths using width-wise streamlining | VGG19 · CIFAR-100 | Top-1 accuracy: 65.53 ± 0.66 -> 66.99 ± 0.52 | Helped |
| Adjust layer widths using width-wise streamlining | ResNet18 · CIFAR-10 | Top-1 accuracy: 92.99 ± 0.23 -> 93.60 ± 0.13 | Helped |
| Adjust layer widths using width-wise streamlining | ResNet18 · CIFAR-100 | Top-1 accuracy: 72.84 ± 0.27 -> 73.38 ± 0.30 | Helped |
| Adjust layer widths using width-wise streamlining | ResNet50 · CIFAR-10 | Top-1 accuracy: 92.47 ± 0.43 -> 93.12 ± 0.33 | Helped |
| Adjust layer widths using width-wise streamlining | ResNet50 · CIFAR-100 | Top-1 accuracy: 73.23 ± 0.45 -> 73.53 ± 0.50 | Helped |
| Adjust layer widths using width-wise streamlining | VGG16 · TinyImageNet validation | Accuracy: 49.78 ± 0.92 -> 54.40 ± 0.38 | Helped |
| Adjust layer widths using width-wise streamlining | ResNet18 · TinyImageNet validation | Accuracy: 54.46 ± 0.41 -> 56.56 ± 0.27 | Helped |
| Increase the second-layer width from 16 to 64 (ablation) | GCN · Cora | Accuracy: 76.20% -> 80.30% | Helped |
| Increase the second-layer width from 64 to 128 (ablation) | GCN · Cora | Accuracy: 80.30% -> 79.80% | No clear effect |
Surprising findings
- Layers where neurons can be merged (IFM) show the IS pattern, while layers with no mergeable neurons show DS.
- Weight norm at epoch 6 correlates weakly (0.4254) with final norm, whereas the norm change between epochs 4 and 6 correlates strongly (0.8891).
Limitations
- Width adjustment is manual, with FLOPs controlled by hand rather than automatically.
- The IS or DS pattern can only be determined at convergence, not early in training.
- Evaluation is limited to supervised learning with CNN architectures on small image datasets.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet cora enwiki8
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet
Metrics: top-1 accuracy number of parameters FLOPs
Baselines: VGG16 VGG19 ResNet18 ResNet50
Builds on: Frequency principle intrinsic dimension IFM neuron merging
Closest papers by meaning
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.87)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.87)
- On the Weight Dynamics of Deep Normalized Networks (ICML 2024 · similarity 0.86)
- Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs (NeurIPS 2025 · similarity 0.86)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.86)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.