Research map / Architectures and efficiency / RNNs and normalization
Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ASI, which selects per-layer activation ranks once before training via a perplexity-based budgeted search and then compresses activations with a single warm-started subspace iteration, cutting activation memory and training FLOPs for on-device learning.
Problem
Activation memory and computation of backpropagation limit on-device learning; prior low-rank activation compression (HOSVD with explained variance) has prohibitive per-iteration decomposition cost and no hard memory control.
Approach
Replace per-iteration HOSVD with a single warm-started subspace iteration on each unfolded activation mode, and choose ranks offline once before training using a gradient-error-based perplexity measure under a memory budget, exploiting the short-term stability of activation maps across training steps.
Key results
- ASI reduces activation memory up to 120.09x versus vanilla training at similar accuracy on Pets with MCUNet, and cuts training FLOPs up to 1.86x.
- On Raspberry Pi 5 with MCUNet on CIFAR-10, ASI is 1.56x faster than vanilla training per training step and 91.0x faster than HOSVD.
- On TinyLlama 1B fine-tuning BoolQ with 5 layers, ASI reduces activation memory up to about 2500x with roughly 1.9x fewer TFLOPs while keeping decent accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ASI instead of vanilla training | MCUNet fine-tuning · ImageNet-1k, 2 layers | accuracy: 62.1 -> 61.7 | Hurt |
| Use ASI instead of vanilla training | MCUNet fine-tuning · ImageNet-1k, 2 layers | activation memory (MB): 13.78 -> 0.38 | Helped |
| Use ASI instead of vanilla training | MCUNet fine-tuning · ImageNet-1k, 2 layers | GFLOPs: 19.31 -> 11.01 | Helped |
| Use ASI instead of HOSVD (ε=0.8) | MCUNet fine-tuning · ImageNet-1k, 2 layers | GFLOPs: 1988.05 -> 11.01 | Helped |
| Use ASI instead of vanilla training | TinyLlama 1B fine-tuning · BoolQ, 1 layer | accuracy: 65.94 -> 64.69 | Hurt |
| Use ASI (rank=20) instead of vanilla training | TinyLlama 1B fine-tuning · BoolQ, 5 layers | activation memory (MB): 4352 -> 1.72 | Helped |
| Use ASI (rank=20) instead of vanilla training | TinyLlama 1B fine-tuning · BoolQ, 5 layers | TFLOPs: 15.11 -> 8.31 | Helped |
| Add warm-start to ASI (ablation) | MCUNet fine-tuning · CIFAR-10 | accuracy: +3.87% | Helped |
Surprising findings
- Warm-starting the subspace iteration improves ASI accuracy by 3.87% on average, while even without warm start the model still converges with a single iteration.
- HOSVD's actual memory tends to stay below its peak, so ASI never reaches the HOSVD-derived memory budget in practice.
Limitations
- The backtracking rank-selection search is brute-force and becomes resource-intensive as the number of fine-tuned layers grows.
- Rank selection and perplexity estimation are performed offline once per model, and the method relies on activation stability assumptions across steps.
- Experiments on LLMs are limited to a single model (TinyLlama 1B) with a fixed rank of 20 because HOSVD is infeasible there.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100 CUB200 Flowers102 Pets Pascal VOC12 Cityscapes BoolQ
Benchmarks: CIFAR-10 CIFAR-100 CUB200 Flowers102 Pets ImageNet-1k Pascal VOC12 semantic segmentation BoolQ
Metrics: top-1 accuracy mIoU mAcc activation memory (MB) TFLOPs/GFLOPs latency on Raspberry Pi 5
Baselines: Vanilla training Gradient filtering HOSVD (explained variance 0.8) LoRA variants
Builds on: HOSVD activation compression PowerSGD ASVD gradient filtering AdaLoRA
Closest papers by meaning
- Activation Map Compression through Tensor Decomposition for Deep Learning (NeurIPS 2024 · similarity 0.88)
- TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · similarity 0.85)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
- Maestro: Uncovering Low-Rank Structures via Trainable Decomposition (ICML 2024 · similarity 0.83)
- Mem-T: Densifying Rewards for Long-Horizon Memory Agents (ICML 2026 · similarity 0.83)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.