Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes nonparametric transfer learning (NPTL), which applies Bayesian nonparametric learning with a Dirichlet process prior centered on linear-probed pseudo-labels of downstream inputs, and samples posteriors in parallel via a block Dirichlet bootstrap for BMA.
Problem
Gaussian weight priors centered on pre-trained solutions can hinder downstream BMA under distribution shift between upstream and downstream data, and posterior sampling methods like SGHMC are sequential and hard to parallelize.
Approach
Replace the weight-space prior with a nonparametric Dirichlet process prior over the data distribution, whose base measure uses linear-probed predictions on downstream inputs (no upstream data needed); posterior samples are drawn by weighted SGD on downstream and pseudo data using block Dirichlet weights.
Key results
- On ResNet-20x4, NPTL reaches average ACC 0.765 and NLL 0.838, beating the best baseline Ensemble + L2SP (0.762 / 0.853).
- On ResNet-50 across five datasets, NPTL achieves average ACC 0.854 and NLL 0.554 versus 0.850 and 0.594 for Ensemble + L2SP.
- On CIFAR-10-C, NPTL attains average ACC 0.793 and NLL 0.698, outperforming all baselines at every intensity level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NPTL instead of the strongest reported baseline | ResNet-20x4 BMA · C10, C100, F101, D120 (Avg.) | ACC: 0.762 -> 0.765 | Helped |
| Use NPTL instead of the strongest reported baseline | ResNet-20x4 BMA · C10, C100, F101, D120 (Avg.) | NLL: 0.853 -> 0.838 | Helped |
| Use NPTL instead of the strongest reported baseline | ResNet-50 BMA · B200, C101, D47, F102, P37 (Avg.) | ACC: 0.850 -> 0.854 | Helped |
| Use NPTL instead of the strongest reported baseline | ResNet-50 BMA · B200, C101, D47, F102, P37 (Avg.) | NLL: 0.594 -> 0.554 | Helped |
| Use NPTL instead of the reported ensemble baseline | ViT-B/16 BMA · B200, C101, D47, F102, P37 (Avg.) | ACC: 0.895 -> 0.903 | Helped |
| Use NPTL instead of the reported ensemble baseline | ViT-B/16 BMA · B200, C101, D47, F102, P37 (Avg.) | NLL: 0.400 -> 0.372 | Helped |
| Use NPTL instead of the strongest reported baseline | ResNet-20x4 BMA · CIFAR-10-C (corruption intensities 1–5; Avg.) | ACC: 0.763 -> 0.793 | Helped |
| Use NPTL instead of the strongest reported baseline | ResNet-20x4 BMA · CIFAR-10-C (corruption intensities 1–5; Avg.) | NLL: 0.847 -> 0.698 | Helped |
| Use NPTL instead of the strongest reported baseline | ResNet-20x4 BMA · CIFAR-10-C (corruption intensities 1–5; Avg.) | ECE: 0.076 -> 0.066 | Helped |
| Use NPTL instead of the ensemble baseline | RoBERTa-Base BMA · GLUE (CoLA, MRPC, RTE; Avg.) | GLUE: 0.789 -> 0.797 | Helped |
| Use NPTL instead of the ensemble baseline | RoBERTa-Base BMA · GLUE (CoLA, MRPC, RTE; Avg.) | NLL: 0.411 -> 0.371 | Helped |
| Average weights of NPTL posterior samples (ablation) | NPTL on B200, C101, D47, F102, P37 · Image classification (Avg.) | ACC: 0.854 -> 0.848 | Hurt |
Surprising findings
- NPTL-Soup, a single weight-averaged model, reaches competitive performance with the BMA over NPTL samples at about one-tenth the inference cost.
Limitations
- Theoretical convergence analysis covers only a simplified, non-blocked version of the algorithm, with the implemented blocked version left for future work.
- Ensemble-based BMA incurs multiple forward passes at inference; NPTL-Soup trades some accuracy for efficiency.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Food-101 Stanford Dogs CUB-200 Caltech-101 DTD Oxford Flowers 102 Oxford-IIIT Pet GLUE CoLA GLUE MRPC GLUE RTE
Benchmarks: CIFAR-10 CIFAR-100 Food-101 Stanford Dogs CUB-200 Caltech-101 DTD Oxford Flowers 102 Oxford-IIIT Pet GLUE CoLA GLUE MRPC GLUE RTE
Metrics: accuracy negative log-likelihood expected calibration error GLUE metric
Baselines: SGHMC + L2SP SGHMC + PTYL Ensemble + L2SP Ensemble + PTYL Fine-tune NPTL-Soup
Builds on: Bayesian nonparametric learning L2SP PTYL Stochastic Gradient Hamiltonian Monte Carlo Greedy Soup
Closest papers by meaning
- Transfer Learning Meets Functional Linear Regression: No Negative Transfer Under Posterior Drift (AAAI 2025 · similarity 0.84)
- Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported (NeurIPS 2024 · similarity 0.84)
- Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing (NeurIPS 2026 · similarity 0.83)
- A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning (NeurIPS 2025 · similarity 0.83)
- Smoothness Adaptive Hypothesis Transfer Learning (ICML 2024 · similarity 0.83)
- Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.