Research map / Architectures and efficiency / Pruning and efficient tuning
A Bayesian Model Selection Criterion for Selecting Pretraining Checkpoints
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes downstream free energy, a local Bayesian marginal-likelihood model selection criterion for pretraining checkpoints, and a computable pretraining free energy estimated via localized WBIC, which correlates with fine-tuning accuracy on CIFAR-FS and mini-ImageNet.
Problem
Selecting pretraining checkpoints that adapt well to downstream tasks without access to downstream data or knowledge of the downstream task.
Approach
Defines downstream free energy as the negative log of a local marginal likelihood around a pretraining checkpoint, proves pretraining free energy upper-bounds (controls) the downstream free energy under distribution shift with bounded density ratio, and estimates pretraining free energy via a localized WBIC computed with SGLD on pretraining training loss.
Key results
- On CIFAR-FS, pretraining free energy (WBIC) correlates with finetune transfer accuracy at Pearson -0.820 and with 5-shot accuracy at -0.8901, outperforming geometric complexity (-0.767, -0.443) and neural collapse (-0.632, -0.1875).
- Increasing learning rate, decreasing batch size, and increasing momentum all lower the pretraining WBIC and improve downstream transfer accuracy on ResNet-18 (CIFAR-FS).
- Pretraining train loss often collapses to similar values across settings, making it ineffective at distinguishing fine-tuning behavior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select checkpoints with lower pretraining WBIC | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test full fine-tuning | transfer accuracy | Helped |
| Select checkpoints with lower pretraining WBIC | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test 5-shot fine-tuning | transfer accuracy | Helped |
| Increase pretraining learning rate | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test fine-tuning | pretraining WBIC | Helped |
| Decrease pretraining batch size | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test fine-tuning | pretraining WBIC | Helped |
| Increase pretraining momentum | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test fine-tuning | pretraining WBIC | Helped |
| Use pretraining train loss instead of pretraining WBIC to distinguish checkpoints | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test fine-tuning | checkpoint discrimination | Hurt |
| Use pretraining free energy rather than geometric complexity or neural collapse as the pretraining metric | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test full fine-tuning | Pearson correlation with accuracy: -0.767; -0.632; -\textbf{0.820} | Helped |
| Use pretraining free energy rather than geometric complexity or neural collapse as the pretraining metric | ResNet-18 pretrained on CIFAR-FS · CIFAR-FS meta-test 5-shot fine-tuning | Pearson correlation with accuracy: -0.443; -0.1875; -\textbf{0.8901} | Helped |
Surprising findings
- Pretraining train loss, a common baseline, often converges to similar values and fails to distinguish checkpoints' fine-tuning behavior.
Limitations
- Direct link between downstream free energy and downstream predictive performance holds only for Bayesian fine-tuning.
- Computing pretraining WBIC is expensive and may be intractable for very large models.
- Bound becomes uninformative when pretraining and downstream label supports are disjoint or downstream has more classes than pretraining.
Experimental setup
Datasets: CIFAR-FS mini-ImageNet
Benchmarks: CIFAR-FS meta-test mini-ImageNet meta-test
Metrics: fine-tune transfer accuracy 5-shot transfer accuracy Pearson correlation
Baselines: pretraining train loss geometric complexity neural collapse
Builds on: WBIC local learning coefficient neural collapse geometric complexity
Closest papers by meaning
- BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.87)
- Maximizing Intermediate Checkpoint Value in LLM Pretraining with Bayesian Optimization (ICML 2025 · similarity 0.86)
- Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? (ICML 2025 · similarity 0.86)
- Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining (NeurIPS 2026 · similarity 0.86)
- Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuning (ICLR 2024 · similarity 0.86)
- Model Fusion through Bayesian Optimization in Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.