Research map / Label-efficient and robust learning / Recognition and long-tail
The Tunnel Effect: Building Data Representations in Deep Neural Networks
Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miłoś, Tomasz Trzcinski
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use deeper layers after the representation extractor as a compressing tunnel | Overparameterized image classifiers · CIFAR-10, CIFAR-100, and CINIC-10 | linear-probe accuracy across layers | No clear effect |
| Continue through the tunnel rather than using features at its entry | VGG-19 and ResNet-34 trained on CIFAR-10 · OOD datasets | linear-probe accuracy | Hurt |
| Increase the number of classes in the source task | VGG-19 and ResNet-34 · CIFAR-100 subsets; OOD linear probing | OOD accuracy | Improved |
| Increase network depth while holding the task fixed (ablation) | MLP · CIFAR-10 | extractor length | No clear effect |
| Increase network width (ablation) | VGG networks · CIFAR-10 | tunnel length | Improved |
| Increase network width from 30% to 100% (ablation) | ResNet-34 · CIFAR-10 | layer at 95% of final probing accuracy: 19 { ($56\%)} -> 21 { ($61\%)} | Mixed |
| Train on a dataset with fewer classes (ablation) | VGG-19 and ResNet-34 · CIFAR-100 class subsets | tunnel length | Improved |
| Train on CINIC-10 instead of CIFAR-10, with more training samples but the same number of classes (ablation) | VGG-19 and ResNet-34 · CINIC-10 versus CIFAR-10 | tunnel length | No clear effect |
| Reset tunnel weights to their pre-optimization values (ablation) | VGG-19 · CIFAR-10 | model performance | Hurt |
| Replace the tunnel from one task with the tunnel from another task (ablation) | VGG-19 task-specific extractor-tunnel combinations · CIFAR-10, two five-class tasks | accuracy: 92.04% -> 92.5%; 56.8% -> 58.04 % | No clear effect |
| Attach a linear probe to the extractor instead of the extractor plus tunnel (ablation) | VGG-19 after Task 2 · CIFAR-10, Task 1 | accuracy: 56.1% -> 74.4% | Improved |
| Train a shorter VGG network from scratch (ablation) | VGG-19 · CIFAR-10, sequential two-task learning | catastrophic forgetting | Improved |
Closest papers by meaning
- What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.86)
- The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention (ICML 2022 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.85)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.85)
- Measuring Representational Shifts in Continual Learning: A Linear Transformation Perspective (ICML 2025 · similarity 0.85)
- Towards flexible perception with visual memory (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
