Research map / Architectures and efficiency / RNNs and normalization
Efficient Resource-Constrained Training of Transformers via Subspace Optimization
Le-Trung Nguyen, Enzo Tartaglione, Van-Tam Nguyen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use warm-started subspace iteration (WSI) instead of full SVD at every iteration (ablation) | WASI weight decomposition · ViT Pets | training FLOPs at equal accuracy: 1.36x fewer FLOPs; about 35% higher accuracy at equal FLOPs | Improved |
| raise explained variance threshold from 0.4 to 0.9 (ablation) | WASI · ViT CIFAR-10 (all linear layers) | accuracy (%): 68.99 -> 96.24 | Improved |
| raise explained variance threshold from 0.4 to 0.9 (ablation) | WASI · ViT CIFAR-10 (all linear layers) | training memory (MB): 39.39 -> 179.61 | Hurt |
| apply WASI at epsilon 0.9 instead of vanilla training | ViT · ViT CIFAR-10 (all linear layers) | accuracy (%): 97.32 -> 96.24 | No clear effect |
| apply WASI at epsilon 0.9 instead of vanilla training | ViT · ViT CIFAR-10 (all linear layers) | training memory (MB): 2349.00 -> 179.61 | Improved |
| apply WASI at epsilon 0.9 instead of vanilla training | ViT · ViT CIFAR-10 on Raspberry Pi 5 | training time (s): 23.87 -> 16.57 | Improved |
| compress weights as well as activations (WASI vs ASI) at epsilon 0.9 | ASI · ViT CIFAR-10 on Raspberry Pi 5 | training time (s): 25.52 -> 16.57 | Improved |
| compress weights as well as activations (WASI vs ASI) at epsilon 0.4 | ASI · ViT CIFAR-10 on Raspberry Pi 5 | inference time (s): 7.69 -> 3.15 | Improved |
| apply WSI to last 1 to 4 convolutional layers at small epsilon | MCUNet · Pets | weight memory and accuracy | No clear effect |
| apply WSI to convolutional layers at epsilon 0.9 | MCUNet · Pets | weight memory | Hurt |
| apply WASI at epsilon 0.9 instead of vanilla training | SwinT · multiple datasets | memory and FLOPs: memory by up to 62x; FLOPs by 1.5x | Improved |
| apply WASI instead of vanilla training | TinyLlama · BoolQ | activation and weight memory: 953.86x and 30.12x reduction | Improved |
Closest papers by meaning
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.84)
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.83)
- Taming Transformer Without Using Learning Rate Warmup (ICLR 2025 · similarity 0.83)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.83)
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.83)
- Looped Transformers are Better at Learning Learning Algorithms (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
