Research map / Architectures and efficiency / RNNs and normalization
TinyFoA: Memory Efficient Forward-Only Algorithm for On-Device Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TinyFoA, a backpropagation-free forward-only training method combining joint horizontal and vertical layer-wise training with binary weights and activations to cut on-device training memory.
Problem
Forward-only learning algorithms like Forward-Forward still need too much training memory for mobile edge and IoT devices, and existing binarization memory techniques are designed for backpropagation and degrade forward-only methods.
Approach
Train only a slice of a single layer at a time (layer-wise plus partial per-layer updates via M×M weight slices) with a fixed random auxiliary classifier for local losses, and binarize weights and activations, with gradient clipping through binarized paths; this keeps accuracy close to BP while cutting forward, gradient, and update memory.
Key results
- TinyFoA with joint horizontal/vertical training and binary weights reduces forward-pass training memory by ~96x, gradient memory by ~24x, and update memory by ~12x versus BP while keeping comparable error.
- On MNIST with binary weights and activations, TinyFoA-V has 2.06% error versus 90.47% for FF+BW+BA and 89.94% for PEPITA+BW+BA.
- On CIFAR-10 with binary weights and activations, TinyFoA reaches 50.38% error versus 45.12-ish range of full-precision TinyFoA-V-BA-BW (45.12%) and BP+BW+BA+V 51.32%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TinyFoA instead of BP | Fully connected network training · MNIST | error (%): 1.33±0.04 -> 2.10±0.09 | Hurt |
| Use TinyFoA instead of BP | Fully connected network training · CIFAR-10 | error (%): 43.62±0.33 -> 50.38±0.17 | Hurt |
| Use TinyFoA instead of BP | Fully connected network training · CIFAR-100 | error (%): 72.22±0.43 -> 80.08±0.15 | Hurt |
| Use TinyFoA instead of BP | Fully connected network training · MIT-BIH | error (%): 8.25±0.46 -> 11.69±0.69 | Hurt |
| Use binary weights and binary activations (ablation) | TinyFoA-V · MNIST | error (%): 1.84±0.02 -> 2.06±0.03 | Hurt |
| Add binary weights (ablation) | TinyFoA-V-BA-BW · MNIST | error (%): 1.62±0.08 -> 1.84±0.02 | Hurt |
| Add binary activations (ablation) | TinyFoA-V-BA · MNIST | error (%): 1.84±0.02 -> 2.06±0.03 | Hurt |
| Add vertical layer-wise training (ablation) | TinyFoA · MNIST | error (%): 2.06±0.03 -> 2.10±0.09 | Hurt |
| Use vertical layer-wise training (ablation) | TinyFoA · MNIST | gradient memory (MB): 16.00 -> 4.01 | Helped |
| Use vertical layer-wise training (ablation) | TinyFoA · MNIST | update memory (MB): 32.01 -> 8.00 | Helped |
| Increase slice count M (ablation) | TinyFoA fully connected network · MNIST and CIFAR-10 | error (%) | No clear effect |
| Increase slice count M (ablation) | TinyFoA locally connected network · MNIST and CIFAR-10 | error (%) | Helped |
Surprising findings
- Binarizing activations causes large accuracy collapse for PEPITA and FF (near 90% error on MNIST) while TinyFoA stays near 2% error.
- On CIFAR-100, FF with full precision achieves lower error (85.76%) than some TinyFoA variants in the baseline table, reversing the usual ordering.
Limitations
- Evaluated only on fully connected and locally connected networks; not on Transformers or LLMs.
- Memory numbers are theoretically calculated rather than measured because PyTorch lacks fine-grained memory management.
- Binary weights and activations degrade accuracy for some baselines and TinyFoA on CIFAR datasets relative to BP.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 MIT-BIH Arrhythmia ECG dataset
Benchmarks: MNIST CIFAR-10 CIFAR-100 MIT-BIH Arrhythmia
Metrics: classification error (%) dynamic training memory (MB) for forward, gradient, update computation (MB)
Baselines: BP DRTP PEPITA FF PEPITA-TL
Builds on: Direct Random Target Projection PEPITA Forward-Forward Binarized Neural Networks Feedback alignment
Closest papers by meaning
- TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge (ICML 2024 · similarity 0.85)
- TinyTTA: Efficient Test-time Adaptation via Early-exit Ensembles on Edge Devices (NeurIPS 2024 · similarity 0.84)
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning (ICML 2025 · similarity 0.82)
- Stepping Forward on the Last Mile (NeurIPS 2024 · similarity 0.82)
- The Price of Locality: Why Forward-Forward Underperforms Backpropagation? (NeurIPS 2026 · similarity 0.81)
- AtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.