Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Quick-Tune jointly selects a pretrained image classifier and its finetuning hyperparameters via meta-learned gray-box Bayesian optimization with a cost-aware acquisition, using a 20k-configuration meta-dataset over 24 timm models and Meta-Album datasets.
Problem
Practitioners must choose which pretrained model to finetune from a growing model hub and how to set its finetuning hyperparameters, and trial-and-error is time-consuming and suboptimal.
Approach
Combine combined algorithm selection and hyperparameter optimization (CASH) for finetuning with gray-box learning-curve prediction, meta-learned deep-kernel GP and MLP cost estimator from a large meta-dataset of finetuning runs, and a cost-sensitive expected improvement acquisition.
Key results
- Quick-Tune achieves normalized regret of 0.052 on Meta-Album extended versus 0.199-0.225 for default-hyperparameter single models (XCiT, BEiT, DLA).
- Quick-Tune outperforms ASHA, BOHB, DEHB, DyHPO and random search on normalized regret across all three Meta-Album versions over 1, 4 and 16 hour budgets.
- On the 24-hour budget, Quick-Tune reaches normalized regret 0.018 (micro), 0.012 (mini) and 0.003 (extended), outperforming DINOv2 + LoRA and DINOv2 linear probing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly search pretrained models and finetuning hyperparameters instead of using fixed default configurations | Pretrained image classifiers · Meta-Album Micro, Mini, and Extended | normalized regret: BEiT+Default HP: 0.229_{\pm 0.081}, 0.281_{\pm 0.108}, 0.225_{ \pm 0.059 }; XCiT+Default HP: 0.223_{\pm 0.075}, 0.290_{\ | Helped |
| Use Quick-Tune CASH on a model hub instead of finetuning DINOv2 with LoRA | DINOv2 and Quick-Tune · Meta-Album Micro, Mini, and Extended at 4 and 24 hours | normalized regret: QuickTune: 0.072_{\pm 0.024}, 0.039_{\pm 0.014}, 0.042_{\pm 0.016}; Dinov2 + LoRA: 0.541_{\pm 0.093}, 0.049_{\pm 0.018}, | Helped |
| Use Quick-Tune CASH on a model hub instead of DINOv2 linear probing | DINOv2 and Quick-Tune · Meta-Album Micro, Mini, and Extended at 4 and 24 hours | normalized regret: QuickTune: 0.072_{\pm 0.024}, 0.039_{\pm 0.014}, 0.042_{\pm 0.016}; Dinov2 + Linear Probing: 0.081_{\pm 0.041}, 0.067_{\ | Helped |
| Include meta-learning of the performance and cost predictors (ablation) | Quick-Tune · Meta-Album Micro, Mini, and Extended | normalized regret | Helped |
| Include cost-awareness in the acquisition function (ablation) | Quick-Tune · Meta-Album Micro, Mini, and Extended | normalized regret | Helped |
| Use a diverse hub of 24 or 15 models instead of a small or single-model pool (ablation) | Quick-Tune · Meta-Album Micro, Mini, and Extended | normalized regret | Helped |
| Jointly optimize model selection and hyperparameters rather than use a two-stage approach | Quick-Tune · Meta-Album | normalized regret | Helped |
Surprising findings
- No single pretrained model ranks optimally across datasets even when its hyperparameters are tuned per dataset, indicating no silver bullet model.
Limitations
- Evaluated only on image classification with timm models and Meta-Album datasets.
- Some large-model runs (e.g., DINOv2 with LoRA) failed due to GPU memory limits, reducing the comparison set.
Experimental setup
Datasets: Meta-Album micro Meta-Album mini Meta-Album extended Imagenette iNaturalist
Benchmarks: Meta-Album micro Meta-Album mini Meta-Album extended Imagenette iNaturalist
Metrics: normalized regret rank
Baselines: BEiT default HP XCiT default HP DLA default HP ASHA BOHB DEHB DyHPO Random Search LogME LEEP NCE DINOv2 + LoRA
Builds on: Expected Improvement deep-kernel Gaussian process DyHPO Meta-Album timm
Closest papers by meaning
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.86)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.86)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.86)
- LEAD: Exploring Logit Space Evolution for Model Selection (CVPR 2024 · similarity 0.86)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.86)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.