Research map / Learning theory and algorithms / Matrix and tensor methods
Low Dimensional Trajectory Hypothesis is True: DNNs can be Trained in Tiny Subspaces
Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, Xiaolin Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train in 15D DLDR subspace instead of all parameters | ResNet8 · CIFAR-10 | accuracy (%): 83.84 -> 85.18 | Improved |
| extract subspace from training trajectory (DLDR) instead of random projection | ResNet8 subspace training · CIFAR-10 | accuracy (%): 70.26 -> 85.18 | Improved |
| use random-projection subspace of Li et al. instead of DLDR (ablation) | ResNet8 subspace training · CIFAR-10 | accuracy (%): 85.18 -> 58.35 | Hurt |
| train in 40D DLDR subspace (P-SGD, 100 sample epochs) instead of 100 epochs SGD sampling | VGG11_bn · CIFAR-100 | accuracy (%): 59.90 -> 70.18 | Improved |
| sample trajectory for 100 epochs instead of 50 for DLDR (ablation) | P-SGD VGG11_bn · CIFAR-100 | accuracy (%): 68.72 -> 70.18 | Improved |
| train in 40D DLDR subspace instead of full SGD for 200 epochs | NasNet · CIFAR-100 | accuracy (%): 77.34 -> 77.03 | No clear effect |
| train in 40D subspace instead of full SGD for 200 epochs | DenseNet121 · CIFAR-100 | accuracy (%): 76.76 -> 76.34 | No clear effect |
| replace SGD with P-BFGS in 40D subspace | ResNet20 · CIFAR-10 | accuracy (%): 91.55 -> 91.72 | No clear effect |
| replace SGD with P-BFGS in 40D subspace | ResNet32 · CIFAR-100 | accuracy (%): 68.40 -> 69.90 | Improved |
| replace SGD with P-BFGS in 120D subspace | ResNet18 · ImageNet | accuracy (%): 69.794 -> 69.720 | No clear effect |
| use P-BFGS with DLDR sampling instead of SGD | ResNet20/32/18 · training time | accuracy (%): about 30% time saved | Improved |
| train in 40D subspace (d=40) rather than full SGD under label noise c=0.7 | ResNet (CIFAR-10) · CIFAR-10 noise c=0.7 | accuracy (%): 49.0 -> 77.22 | Improved |
Closest papers by meaning
- Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality (JMLR 2020 · similarity 0.85)
- Dimensionality-Driven Learning with Noisy Labels (ICML 2018 · similarity 0.85)
- Neural Networks Perform Sufficient Dimension Reduction (AAAI 2025 · similarity 0.83)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.82)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.82)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
