Research map / Architectures and efficiency / Pruning and efficient tuning
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Maestro, which trains low-rank factorized DNN layers end-to-end using LoD (ordered-dropout-style sampling of ranks over factorized weights) with hierarchical group lasso and progressive rank shrinking, showing SVD/PCA recovery in linear cases and competitive accuracy-compute on CIFAR-10, ImageNet and Multi30k.
Problem
Training and deploying large DNNs is costly; existing low-rank methods rely on expensive iterative SVD decompositions or full-rank warm-up and manual per-layer rank selection, and SVD ignores data distribution.
Approach
Embed low-rank decomposition into training via LoD: sample one rank per factorized layer per step (ordered dropout on factorized weights), add a hierarchical group-lasso penalty to zero redundant ranks, and progressively shrink each layer's maximal rank, so ranks are learned per layer from data rather than by SVD.
Key results
- On CIFAR-10 ResNet-18, Maestro reaches 94.19% accuracy at 4.08M parameters versus 94.17% for Pufferfish at 3.3M parameters.
- On Multi30k, Maestro achieves perplexity 6.90 at 0.248 GMACs and 13.80M params versus Pufferfish's 7.34 at 0.996 GMACs and 26.70M params.
- On ImageNet-1k with full decomposition of ResNet-50, Maestro reaches 71.54% accuracy at 9.2M parameters versus Pufferfish 71.03% at 9.4M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Maestro instead of Pufferfish | Transformer · Multi30k | perplexity: 7.34{ $\pm0.12$} -> 6.90{ $\pm0.07$} | Helped |
| Use Maestro instead of Pufferfish | ResNet-50 with all layers decomposed · ImageNet-1k | accuracy: 71.03 -> 71.54 | Helped |
| Remove hierarchical group lasso (ablation) | Maestro ResNet-18 · CIFAR-10 | accuracy: 94.19{ $\pm$0.39} -> 94.04{ $\pm$0.10} | No clear effect |
| Remove hierarchical group lasso (ablation) | Maestro ResNet-18 · CIFAR-10 | Rel. GMACs (Train.): 1.00$\times$ -> 1.33$\times$ | Hurt |
| Remove hierarchical group lasso (ablation) | Maestro ResNet-18 · CIFAR-10 | Params. (M): 4.08{ $\pm$0.020} -> 11.2{ $\pm$0.000} | Hurt |
| Remove progressive shrinking (ablation) | Maestro ResNet-18 · CIFAR-10 | accuracy: 94.19{ $\pm$0.39} -> 94.12{ $\pm$0.36} | No clear effect |
| Remove progressive shrinking (ablation) | Maestro ResNet-18 · CIFAR-10 | Rel. GMACs (Train.): 1.00$\times$ -> 1.33$\times$ | Hurt |
| Add an extra full low-rank pass at each training step (ablation) | Maestro ResNet-18 · CIFAR-10 | accuracy: 94.19{ $\pm$0.39} -> 94.05{ $\pm$0.32} | No clear effect |
| Add an extra full low-rank pass at each training step (ablation) | Maestro ResNet-18 · CIFAR-10 | Rel. GMACs (Train.): 1.00$\times$ -> 1.97$\times$ | Hurt |
Surprising findings
- SVD-factorized models with greedy pruning perform significantly worse than the learned Maestro decomposition, indicating SVD ordering is not data-aware.
- Sampling only one rank per step matches the quality of an extra full-rank pass, which the authors report makes that variant about twice as computationally demanding without accuracy gain.
Limitations
- No guarantee on final model footprint; controlled only indirectly via lambda_gl and epsilon_ps.
- Only CNN and Transformer layers evaluated; recurrent layers not explored.
- Uniform rank sampling may slow rank exploration; theoretical guarantees limited to linear models.
Experimental setup
Datasets: MNIST CIFAR-10 ImageNet-1k Multi30k
Benchmarks: CIFAR-10 ImageNet-1k Multi30k
Metrics: accuracy perplexity GMACs parameter count
Baselines: IMP RareGems XNOR-Net Spectral Initialisation Pufferfish Cuttlefish
Builds on: Ordered Dropout Hierarchical Group Lasso Pufferfish Eckart-Young-Mirsky theorem
Closest papers by meaning
- Efficient Learning with Sine-Activated Low-Rank Matrices (ICLR 2025 · similarity 0.86)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.85)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.85)
- Lambda-Skip Connections: the architectural component that prevents Rank Collapse (ICLR 2025 · similarity 0.85)
- Position: A Theory of Deep Learning Must Include Compositional Sparsity (ICML 2025 · similarity 0.85)
- Low Rank Gradients and Where to Find Them (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.