Research map / Label-efficient and robust learning / Recognition and long-tail
Adaptive Random Feature Regularization on Fine-tuning Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaRand, a regularizer for fine-tuning that pulls features toward random reference vectors sampled from adaptively updated class-conditional Gaussian priors, requiring no source information.
Problem
Fine-tuning deep networks overfits on small target datasets; existing regularizers either need source labels/datasets or add heavy computation, and the fixed class-agnostic random regularization (RandReg) is sensitive to prior choice and shrinks feature norm and entropy.
Approach
Replace RandReg's fixed class-agnostic prior with class-conditional Gaussian priors initialized from pre-trained target statistics and updated during training via an EMA of running class-mean features plus intra-class and inter-class distance losses, so features keep norm and entropy and form separable clusters.
Key results
- On Cars with ResNet-50 (ImageNet classification), AdaRand reaches 91.17% top-1 accuracy versus 90.82% for UOT and 90.66% for Co-Tuning, while RandReg-best reaches 90.61%.
- With only 10% of Cars training data, AdaRand achieves 27.55% versus 19.58% for fine-tuning and 26.15% for the best RandReg prior.
- AdaRand is the best method on 5 of 6 datasets (Pets excepted), with 74.60% on Aircraft and 96.68% on Flowers, and it is best across all pre-training and architecture combinations in Table 2 (e.g., CLIP ViT-B/16 92.84%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AdaRand instead of RandReg-N(μs, σs²) | ResNet-50 fine-tuning · Cars | top-1 test accuracy (%): 90.61^{\pm \text {.24}} -> 91.17^{\pm \textbf {.13}} | Helped |
| use AdaRand instead of the strongest listed baseline | ResNet-50 fine-tuning · Aircraft | top-1 test accuracy (%): 73.89^{\pm \text {.10}} -> 74.60^{\pm \textbf {.10}} | Helped |
| use AdaRand instead of RandReg-N(0, 1) | ResNet-50 fine-tuning · Birds | top-1 test accuracy (%): 80.48^{\pm \text {.06}} -> 81.27^{\pm \textbf {.26}} | Helped |
| use AdaRand instead of RandReg-N(0, 1) | ResNet-50 fine-tuning · DTD | top-1 test accuracy (%): 72.57^{\pm \text {.26}} -> 74.86^{\pm \textbf {.22}} | Helped |
| use AdaRand instead of RandReg-U(0, 1) | ResNet-50 fine-tuning · Flower | top-1 test accuracy (%): 96.02^{\pm \text {.05}} -> 96.68^{\pm \textbf {.14}} | Helped |
| use AdaRand instead of DR-Tune | ResNet-50 fine-tuning · Pets | top-1 test accuracy (%): 93.24^{\pm \textbf {.11}} -> 92.34^{\pm \text {.26}} | Hurt |
| use AdaRand instead of RandReg-CP | ResNet-50 fine-tuning · Cars, 10% training data | top-1 test accuracy (%): 26.45^{\pm .26} -> 27.55^{\pm \textbf {.10}} | Helped |
| remove ℓintra (ablation) | AdaRand · Cars | top-1 test accuracy (%): 91.17^{\pm \textbf {.13}} -> 90.81^{\pm \text {.21}} | Hurt |
| remove ℓinter (ablation) | AdaRand · Cars | top-1 test accuracy (%): 91.17^{\pm \textbf {.13}} -> 90.99^{\pm \text {.06}} | Hurt |
| use a fixed conditional prior rather than adaptively updating it (ablation) | AdaRand · Cars | top-1 test accuracy (%): 90.55^{\pm \text {.17}} -> 91.17^{\pm \textbf {.13}} | Hurt |
| use AdaRand instead of RandReg across pre-training methods and architectures | Fine-tuning models · Cars | top-1 test accuracy (%) | Helped |
| use adaptive class-conditional priors instead of RandReg's fixed class-agnostic prior | Fine-tuning models · Cars | feature norm and entropy | Helped |
Surprising findings
- RandReg with a fixed prior underperforms plain L2 feature norm penalty (FNP) on SimCLR features, showing that naive randomness can hurt fine-tuning.
- RandReg shrinks feature norm and entropy during fine-tuning, limiting cross-entropy gradients and mutual information.
Limitations
- Gains over the strongest baselines are modest on some datasets (e.g., Pets where DR-Tune is best), and results are reported on a limited set of datasets and architectures.
- Hyperparameters such as EMA decay alpha=0.5 and lambda=1.0 were fixed; their effects are only discussed in the supplement.
- Method is evaluated only on discriminative image classification; extension to generative tasks is left as future work.
Experimental setup
Datasets: ImageNet Aircraft Birds Cars DTD Flowers Pets CIFAR-10
Benchmarks: Aircraft Birds Cars DTD Flowers Pets CIFAR-10 (PCA visualization)
Metrics: top-1 test accuracy training time per epoch GPU memory usage feature norm feature entropy mutual information gradient norm
Baselines: Fine-tuning FNP L2SP DELTA BSS Core-tuning DR-Tune Co-Tuning UOT RandReg
Builds on: RandReg feature norm penalty (FNP) Co-Tuning DR-Tune L2SP
Closest papers by meaning
- AdaFilter: Adaptive Filter Fine-Tuning for Deep Transfer Learning (AAAI 2020 · similarity 0.86)
- Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guarantees (ICML 2022 · similarity 0.86)
- Deep Semi-Random Features for Nonlinear Function Approximation (AAAI 2018 · similarity 0.85)
- Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability (AAAI 2024 · similarity 0.84)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.83)
- AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.