Research map / Architectures and efficiency / Pruning and efficient tuning
Study of Training Dynamics for Memory-Constrained Fine-Tuning
Aël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen, Enzo Tartaglione
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| resample channels dynamically between epochs instead of static selection (ablation) | channel selection strategies · CNN fine-tuning on 7 datasets | top-1 accuracy | Improved |
| restrict random channel sampling to top-K layers by RGN (ablation) | D-Full Random · MbV2-w0.35, budget 27 946, 7-dataset average | top-1 accuracy: 74.24 -> 74.91 | Improved |
| restrict random channel sampling to top-K layers by RGN (ablation) | D-Full Random · MbV2-w0.35, budget 66 592, 7-dataset average | top-1 accuracy: 75.09 -> 75.62 | Improved |
| random channel sampling in top-K layers instead of deterministic highest-RGN selection (ablation) | TraDy · CNNs across datasets | top-1 accuracy: 74.49 -> 74.91 (MbV2-w0.35, 27 946) | Improved |
| use TraDy instead of Sparse Update (SU) | MbV2-w0.35 · 7-dataset average, budget 27 946 | top-1 accuracy: 74.22 -> 74.91 | Improved |
| use TraDy instead of Velocity on larger budgets | MbV2-w0.35 · MbV2-w0.35, 7-dataset average, budget 66 592 | top-1 accuracy: 75.73 vs 75.62 (budget 66 592) | Hurt |
| vary number K of top layers for random sampling (ablation) | TraDy on MobileNetV2 CIFAR-10 · smallest budget | top-1 accuracy | Mixed |
| use reweighted gradient norm instead of raw gradient norm for channel thresholding (ablation) | gradient-threshold freezing · MobileNetV2 CIFAR-10 | accuracy vs memory | Improved |
| channel-level selection instead of neuron-level (Velocity) | activation sparsity · MobileNetV2 Food | activation sparsity: 20-40% -> 97-99% | Improved |
Closest papers by meaning
- Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks (ICML 2026 · similarity 0.85)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.85)
- RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning (NeurIPS 2026 · similarity 0.84)
- SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity (CVPR 2025 · similarity 0.84)
- Beyond Random: Automatic Inner-loop Optimization in Dataset Distillation (NeurIPS 2025 · similarity 0.84)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
