Research map / Architectures and efficiency / Pruning and efficient tuning
TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TinyTrain, an on-device training method that meta-trains a backbone with ProtoNet on MiniImageNet and then dynamically selects layers/channels to update at deployment using a Fisher-based, memory- and compute-aware criterion.
Problem
On-device training of DNNs on memory- and compute-constrained edge/MCU devices with very few labelled user samples in cross-domain settings, where prior methods lose accuracy, need long training, or rely on expensive offline searches.
Approach
Combine few-shot meta-training (ProtoNet on MiniImageNet) with a task-adaptive sparse update: at deployment, score each layer by Fisher potential normalised by parameter count and MACs, select layers under memory/compute budgets, then pick top-K channels by Fisher information, avoiding offline evolutionary search.
Key results
- TinyTrain averages 69.3% (MCUNet), 65.6% (MobileNetV2), 68.3% (ProxylessNASNet) across nine cross-domain datasets, outperforming FullTrain by 3.6-5.0 pp and SparseUpdate by 2.6-7.7 pp.
- Backward-pass memory is reduced by 1.59-2.23x versus SparseUpdate and compute by 1.52-1.82x, with TinyTrain at 0.78-1.06 MB.
- On Pi Zero 2, TinyTrain runs end-to-end training in about 10 minutes versus two hours for FullTrain; dynamic selection takes 20-35 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TinyTrain's few-shot meta-training to pre-training (ablation) | MCUNet on cross-domain tasks · Meta-Dataset nine-dataset average | top-1 accuracy: +5.6 pp on average | Helped |
| use dynamic rather than L2-Norm-based static channel selection (ablation) | TinyTrain with MCUNet · cross-domain tasks | top-1 accuracy: 0.8-1.7 pp | Helped |
| use dynamic rather than random static channel selection (ablation) | TinyTrain with MCUNet · cross-domain tasks | top-1 accuracy: 1.9-2.5 pp | Helped |
| use Fisher-only layer selection instead of L2-Norm-based layer selection (ablation) | TinyTrain with MCUNet · Meta-Dataset nine-dataset average | top-1 accuracy: 67.9 -> 69.2 | Helped |
| use Fisher-only layer selection instead of L2-Norm-based layer selection (ablation) | TinyTrain with MobileNetV2 · Meta-Dataset nine-dataset average | top-1 accuracy: 62.5 -> 64.3 | Helped |
| use Fisher-only layer selection instead of L2-Norm-based layer selection (ablation) | TinyTrain with ProxylessNASNet · Meta-Dataset nine-dataset average | top-1 accuracy: 62.6 -> 68.2 | Helped |
| include both memory and compute costs alongside Fisher information in the layer-selection criterion (ablation) | TinyTrain with MCUNet · Meta-Dataset nine-dataset average | top-1 accuracy: 68.6 -> 69.3 | Helped |
| include both memory and compute costs alongside Fisher information in the layer-selection criterion (ablation) | TinyTrain with MobileNetV2 · Meta-Dataset nine-dataset average | top-1 accuracy: 63.5 -> 65.7 | Helped |
| include both memory and compute costs alongside Fisher information in the layer-selection criterion (ablation) | TinyTrain with ProxylessNASNet · Meta-Dataset nine-dataset average | top-1 accuracy: 67.5 -> 68.3 | Helped |
| use TinyTrain rather than FullTrain | MCUNet · Meta-Dataset nine-dataset average | top-1 accuracy: 66.9 -> 69.3 | Helped |
| use TinyTrain rather than SparseUpdate | MCUNet · Meta-Dataset nine-dataset average | top-1 accuracy: 64.3 -> 69.3 | Helped |
| use TinyTrain rather than SparseUpdate | TinyTrain versus SparseUpdate on edge-device backward passes · MCUNet, MobileNetV2, and ProxylessNASNet | memory: 1.59-2.23× less memory | Helped |
Surprising findings
- Full fine-tuning (FullTrain) does not guarantee higher accuracy than sparse updates in the few-shot cross-domain setting.
- Fisher-only layer scoring performs nearly as well as the full multi-objective criterion on MCUNet (69.2 vs 69.3).
Limitations
- Evaluation limited to CNN architectures on vision tasks.
- Offline pre-training and meta-training require a server GPU (Quadro RTX 8000).
- FullTrain and TinyTL baselines use batch size 100 while others use batch size 1, complicating memory comparisons.
Experimental setup
Datasets: ImageNet MiniImageNet Traffic Sign Omniglot Aircraft VGG Flowers CUB-200-2011 DTD QuickDraw Fungi COCO
Benchmarks: Meta-Dataset (nine cross-domain datasets)
Metrics: top-1 accuracy backward-pass memory (MB) backward MAC count end-to-end latency (s) energy (kJ)
Baselines: None FullTrain LastLayer TinyTL SparseUpdate Transductive AdapterDrop
Builds on: SparseUpdate (MCUNetV3) ProtoNet TinyTL Meta-Dataset
Closest papers by meaning
- POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging (ICML 2022 · similarity 0.90)
- TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices (NeurIPS 2024 · similarity 0.88)
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning (ICML 2025 · similarity 0.85)
- AtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints (AAAI 2025 · similarity 0.85)
- TinyFoA: Memory Efficient Forward-Only Algorithm for On-Device Learning (AAAI 2025 · similarity 0.85)
- DEX: Data Channel Extension for Efficient CNN Inference on Tiny AI Accelerators (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.