Research map / Architectures and efficiency / RNNs and normalization
Activation Map Compression through Tensor Decomposition for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses stored activation maps for backpropagation using SVD and HOSVD with explained-variance-controlled truncation, reporting large activation-memory savings on CNN fine-tuning with comparable accuracy.
Problem
Backpropagation on edge devices is limited by the memory footprint of stored activation maps needed to compute weight gradients.
Approach
Store only the truncated low-rank components of each layer's input activation (SVD on reshaped matrix, or HOSVD/Tucker across all modes), selected by a cumulative explained-variance threshold, and compute weight gradients directly from these factors via a sequence of smaller convolutions, with a per-layer SNR-based error analysis.
Key results
- HOSVD (eps=0.8) on ResNet18 fine-tuning all four-layer setup reduces activation memory to about 0.97 MB peak for 2 layers vs 12.25 MB for vanilla training at 69.2% vs 69.9% accuracy on ImageNet-1k.
- On MobileNetV2 with setup B, HOSVD (eps=0.8) with 4 layers reaches 63.9% accuracy at 0.73 MB peak versus 65.8% vanilla at 28.71 MB, and gradient filtering at 65.2% with 9.38 MB.
- HOSVD is reported to be on average 4.29 times slower than vanilla training in total training time on a single CIFAR-10 batch, since forward decomposition overhead is high even though backward is faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress activations with HOSVD at ε=0.8 instead of vanilla training | MCUNet · ImageNet-1k, 2 layers | peak memory (MB): 13.78 -> 0.48 | Helped |
| Compress activations with HOSVD at ε=0.8 instead of vanilla training | MCUNet · ImageNet-1k, 2 layers | accuracy: 62.1 -> 61.7 | Hurt |
| Increase retained explained variance from ε=0.8 to ε=0.9 (ablation) | HOSVD on MCUNet · ImageNet-1k, 2 layers | accuracy: 61.7 -> 62.0 | Helped |
| Increase retained explained variance from ε=0.8 to ε=0.9 (ablation) | HOSVD on MCUNet · ImageNet-1k, 2 layers | peak memory (MB): 0.48 -> 1.32 | Hurt |
| Compress activations with HOSVD at ε=0.8 instead of vanilla training | PSPNet · Pascal-VOC12, 5 layers | peak memory (MB): 128.00 -> 0.47 | Helped |
| Compress activations with HOSVD at ε=0.8 instead of vanilla training | PSPNet · Pascal-VOC12, 5 layers | mIoU: 39.36 -> 38.11 | Hurt |
| Increase retained explained variance from ε=0.8 to ε=0.9 (ablation) | HOSVD on PSPNet · Pascal-VOC12, 5 layers | mIoU: 38.11 -> 39.03 | Helped |
Surprising findings
- HOSVD is reported to be on average 4.29x slower than vanilla training in total wall-clock training time despite lower memory.
- Gradient filtering accuracy rises with more fine-tuned layers and then deteriorates, whereas SVD, HOSVD and vanilla keep improving.
Limitations
- Forward-pass decomposition overhead makes total training time several times slower than vanilla training.
- Compression is controlled by explained variance, so memory usage cannot be explicitly bounded and varies during training.
- Experiments are limited to CNN classification and segmentation with fine-tuning; the convergence guarantee is only analyzed via per-layer SNR rather than full convergence proof.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100 CUB200 Flowers102 Pets Cityscapes Pascal-VOC12
Benchmarks: ImageNet-1k CIFAR-10 CIFAR-100 CUB200 Flowers102 Pets Pascal-VOC12
Metrics: top-1 accuracy mIoU mAcc peak activation memory (MB) mean activation memory (MB)
Baselines: Vanilla training Gradient Filtering SVD activation compression
Builds on: Gradient Filtering Lin et al. sub-network fine-tuning (tiny training) Kim et al. Tucker convolution backpropagation
Closest papers by meaning
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning (ICML 2025 · similarity 0.88)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.85)
- Position: Tensor Networks are a Valuable Asset for Green AI (ICML 2024 · similarity 0.85)
- ESPACE: Dimensionality Reduction of Activations for Model Compression (NeurIPS 2024 · similarity 0.85)
- Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks (NeurIPS 2025 · similarity 0.85)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.