AI Paper Atlas

Research map / Learning theory and algorithms / Matrix and tensor methods

Low Dimensional Trajectory Hypothesis is True: DNNs can be Trained in Tiny Subspaces

Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, Xiaolin Huang

TPAMI 2022 · Learning theory and algorithms › Matrix and tensor methods · approach: Geometry of low-rank and SPD structure · 32 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train in 15D DLDR subspace instead of all parametersResNet8 · CIFAR-10accuracy (%): 83.84 -> 85.18Improved
extract subspace from training trajectory (DLDR) instead of random projectionResNet8 subspace training · CIFAR-10accuracy (%): 70.26 -> 85.18Improved
use random-projection subspace of Li et al. instead of DLDR (ablation)ResNet8 subspace training · CIFAR-10accuracy (%): 85.18 -> 58.35Hurt
train in 40D DLDR subspace (P-SGD, 100 sample epochs) instead of 100 epochs SGD samplingVGG11_bn · CIFAR-100accuracy (%): 59.90 -> 70.18Improved
sample trajectory for 100 epochs instead of 50 for DLDR (ablation)P-SGD VGG11_bn · CIFAR-100accuracy (%): 68.72 -> 70.18Improved
train in 40D DLDR subspace instead of full SGD for 200 epochsNasNet · CIFAR-100accuracy (%): 77.34 -> 77.03No clear effect
train in 40D subspace instead of full SGD for 200 epochsDenseNet121 · CIFAR-100accuracy (%): 76.76 -> 76.34No clear effect
replace SGD with P-BFGS in 40D subspaceResNet20 · CIFAR-10accuracy (%): 91.55 -> 91.72No clear effect
replace SGD with P-BFGS in 40D subspaceResNet32 · CIFAR-100accuracy (%): 68.40 -> 69.90Improved
replace SGD with P-BFGS in 120D subspaceResNet18 · ImageNetaccuracy (%): 69.794 -> 69.720No clear effect
use P-BFGS with DLDR sampling instead of SGDResNet20/32/18 · training timeaccuracy (%): about 30% time savedImproved
train in 40D subspace (d=40) rather than full SGD under label noise c=0.7ResNet (CIFAR-10) · CIFAR-10 noise c=0.7accuracy (%): 49.0 -> 77.22Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy