Research map / Architectures and efficiency / RNNs and normalization
Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model depth | CIFAR ResNets · CIFAR-10 and CIFAR-100 | test accuracy: 93.5 -> 94.4; 71.2 -> 73.9 | Helped |
| Increase model width | CIFAR ResNets · CIFAR-10 and CIFAR-100 | test accuracy: 92.0 -> 96.0; 67.8 -> 80.2 | Helped |
| Increase model width or depth, raising capacity relative to training data | ResNets trained on CIFAR-10 and CIFAR-100 · Hidden-layer representations | CKA block-structure emergence | Helped |
| Reduce training-set size, increasing relative model capacity | CIFAR ResNets · Hidden-layer representations | CKA block-structure emergence | Helped |
| Remove the first principal component from representations (ablation) | ResNets with block structure · Hidden-layer representations | CKA block structure | Hurt |
| Prune blocks from the middle stage while keeping residual connections intact (ablation) | ResNets with block structure · CIFAR test set | accuracy | No clear effect |
| Compare wide and deep architectures at similar overall accuracy | CIFAR-10 ResNet-62 (1×) and ResNet-14 (2×) · CIFAR-10 test | mean accuracy: 94.09 ± 0.01 vs. 94.08 ± 0.01; p = 0.47 | No clear effect |
| Compare wide and deep architectures | ImageNet ResNets · ImageNet, example-level predictions | accuracy across examples: 78.00 ± 0.01; 77.97 ± 0.01 | Mixed |
| Increase width rather than depth in the third stage | ImageNet ResNet-50 · ImageNet, structure/geological formation classes | accuracy: 74.9% ± 0.05 vs. 74.6% ± 0.06 | Helped |
| Increase depth rather than width in the third stage | ImageNet ResNet-50 · ImageNet, consumer goods classes | accuracy: 72.4% ± 0.07 vs. 72.1% ± 0.06 | Helped |
| Compare wide and deep architectures | ImageNet ResNet-50 variants · ImageNet, dog classes | accuracy: 81.6%; p = 0.48 | No clear effect |
| Compare representations across random seeds and architectures | Wide and deep ResNets · Hidden-layer representations | CKA similarity within block structure | Hurt |
Closest papers by meaning
- When Representations Align: Universality in Representation Learning Dynamics (ICML 2024 · similarity 0.88)
- Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance (NeurIPS 2024 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.87)
- Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · similarity 0.86)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.