Research map / Probabilistic and scientific modeling / Bayesian optimization
Model Fusion through Bayesian Optimization in Language Model Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BOMF, a model fusion method that uses multi-objective Bayesian optimization over loss and metrics to choose weighted-averaging coefficients for fusing single-trajectory checkpoints, plus BO-based hyperparameter search on lightweight frozen/LoRA proxies.
Problem
Choosing a single good fine-tuned model for pre-trained language models is hard: hyperparameter sensitivity is high, and loss and metric landscapes are misaligned, so uniform weight averaging (SWA, Model Soups) and loss-only weighted averaging underperform.
Approach
Fuse checkpoints from one training trajectory with weights chosen by multi-objective Bayesian optimization (qNEHVI) that jointly minimizes validation loss and metrics, and find hyperparameters with a two-stage BO procedure that can use cheaper frozen-layer or low-rank LoRA proxies because their optimal hyperparameters align with full models.
Key results
- On RoBERTa-base, BOMF reaches 81.40 on RTE and 93.90 F1 on MRPC, improving over HPBO (78.57 RTE, 92.78 MRPC) and the SWA variants on most GLUE subsets.
- On T5-base SQuAD2.0, BOMF achieves 81.82 F1 / 76.21 EM versus 78.28/73.29 for HPBO (full).
- On LLaMA-2-7B SAMSum, BOMF averages 42.03 (Rank 4 LoRA) and 42.19 (full-trajectory variant) versus 41.73 for HPBO (Rank 64).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BOMF instead of Learned SWA | T5-base · SQuAD2.0 | average of F1 and exact match: 78.13 -> 78.31 | Helped |
| optimize F1 and exact match jointly rather than F1 only (ablation) | BOMF on T5-base · SQuAD2.0 | exact match: 75.09 -> 75.79 | Helped |
| optimize F1 and exact match jointly rather than EM only (ablation) | BOMF on T5-base · SQuAD2.0 | F1: 80.40 -> 80.82 | Helped |
| use BOMF instead of HPBO | RoBERTa-base · QNLI | accuracy: 93.01 -> 94.83 | Helped |
| use BOMF instead of HPBO | LLaMA-2-7B · SAMSum | ROUGE-L: 44.33 -> 44.40 | Helped |
| use BOMF instead of Learned SWA | LLaMA-2-7B · SAMSum | ROUGE-2: 28.97 -> 28.61 | Hurt |
| use BOMF instead of the strongest single-objective baseline (ablation) | T5-base · SQuAD2.0 | F1 and exact match: F1: 81.01 -> 80.82; EM: 75.87 -> 75.79 | Mixed |
| search hyperparameters with a lightweight model using 6 frozen layers instead of full-model search (ablation) | RoBERTa-base hyperparameter search · RTE | accuracy: 78.57 -> 78.49 | No clear effect |
Surprising findings
- Loss and metric landscapes are substantially misaligned in PLMs; Spearman correlation between loss and metric is only 0.14 to 0.23 for RoBERTa, T5 and LLaMA, versus 0.62 to 0.66 for vision models.
- Uniform averaging (SWA) can reach good loss but degrade metric performance compared with the best single checkpoint, and the hyperparameters found on frozen-layer or low-rank LoRA proxies transfer to full models.
Limitations
- Quantization-based low-rank methods (e.g., QLoRA) were observed to be hard to aggregate with averaging methods.
- Theoretical explanation of hyperparameter alignment is left open.
- Gains over baselines are often small (e.g., differences of a few tenths of a point on several GLUE tasks) and results come from single configurations.
Experimental setup
Datasets: GLUE (RTE, MRPC, SST-2, QNLI, QQP, MNLI) SQuAD 2.0 SAMSum E2E KorMedMCQA
Benchmarks: GLUE (RTE, MRPC, SST-2, QNLI, QQP, MNLI) SQuAD 2.0 SAMSum KorMedMCQA E2E
Metrics: accuracy F1 exact match ROUGE-1 ROUGE-2 ROUGE-L BLEU METEOR Spearman rank correlation
Baselines: Grid Fine-Tune HPBO SWA Greedy SWA Learned SWA TWA OTfusion ICL
Builds on: Stochastic Weight Averaging Learned Soups Greedy Soups qNEHVI Model Soups
Closest papers by meaning
- InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models (NeurIPS 2025 · similarity 0.91)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.90)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.89)
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.89)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.89)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.