Research map / Architectures and efficiency / Pruning and efficient tuning
TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning
Han Cai, Chuang Gan, Ligeng Zhu, Song Han
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add lite residual modules to bias-only tuning (ablation) | TinyTL-B · Cars | top-1 accuracy (%): 73.4 -> 85.0 | Improved |
| add lite residual modules to bias-only tuning (ablation) | TinyTL-B · Aircraft | top-1 accuracy (%): 63.2 -> 75.4 | Improved |
| add bias tuning to lite residual modules (ablation) | TinyTL-L · Cars | top-1 accuracy (%): 84.2 -> 85.0 | Improved |
| add bias tuning to lite residual modules (ablation) | TinyTL-L · Aircraft | top-1 accuracy (%): 76.4 -> 75.4 | Mixed |
| initialize lite residual modules with pre-trained weights instead of random (ablation) | TinyTL-RandomL+B · Cars | top-1 accuracy (%): 82.4 -> 85.0 | Improved |
| initialize lite residual modules with pre-trained weights instead of random (ablation) | TinyTL-RandomL+B · Flowers | top-1 accuracy (%): 88.0 -> 95.5 | Improved |
| increase resolution from 224 to 320 | TinyTL-L+B · Cars | top-1 accuracy (%): 85.0 -> 88.8 | Improved |
| use TinyTL-L+B instead of fine-tuning last layer only | FT-Last · Cars | top-1 accuracy (%): 50.9 -> 85.0 | Improved |
| use TinyTL-L+B instead of fine-tuning norm layers and last layer | FT-Norm+Last · Training memory | training memory: 192MB -> 37MB | Improved |
| add feature extractor adaptation with last two layers tuned (ablation) | TinyTL FA@352 · Cars | top-1 accuracy (%): 90.7 -> 91.5 | Improved |
| train with batch size 1 instead of 8 (ablation) | TinyTL · Flowers, Aircraft, Cars | top-1 accuracy | No clear effect |
| use TinyTL instead of dynamic activation pruning | Activation pruning · Flowers, Aircraft, Cars | accuracy vs memory | Improved |
Closest papers by meaning
- TinyFoA: Memory Efficient Forward-Only Algorithm for On-Device Learning (AAAI 2025 · similarity 0.87)
- TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · similarity 0.86)
- TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices (NeurIPS 2024 · similarity 0.85)
- DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators (NeurIPS 2024 · similarity 0.84)
- AtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints (AAAI 2025 · similarity 0.84)
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
