Research map / Architectures and efficiency / Pruning and efficient tuning
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Average selected fine-tuned weights instead of choosing the best validation model | ViT-G/14 fine-tuned on ImageNet · ImageNet | top-1 accuracy (%): 90.72 -> 90.94 | Helped |
| Average selected fine-tuned weights instead of choosing the best validation model | ViT-G/14 fine-tuned on ImageNet · ImageNet distribution shifts (average) | average accuracy (%): 84.38 -> 85.02 | Helped |
| Average all fine-tuned weights uniformly rather than greedily selecting ingredients (ablation) | CLIP ViT-B/32 · ImageNet | accuracy (%): 81.03 -> 79.97 | Hurt |
| Use random ingredient order rather than sorting by validation accuracy (ablation) | CLIP ViT-B/32 greedy soup · ImageNet | accuracy (%): 81.03 -> $80.79$ | Hurt |
| Learn soup mixing coefficients on validation data instead of greedy selection (ablation) | CLIP ViT-B/32 · ImageNet | accuracy (%): 81.03 -> 80.89 | Hurt |
| Learn separate mixing coefficients per layer (ablation) | CLIP ViT-B/32 learned soup · ImageNet | accuracy (%): 80.89 -> 81.37 | Helped |
| Average all fine-tuned weights uniformly rather than greedily selecting ingredients (ablation) | CLIP ViT-B/32 · ImageNet distribution shifts | accuracy (%): 47.83 -> 51.45 | Helped |
| Average selected fine-tuned weights rather than selecting the best individual model | BERT · RTE | accuracy: +0.7 | Helped |
| Average selected fine-tuned weights rather than selecting the best individual model | BERT · MRPC | accuracy and F1 average: +0.0 | No clear effect |
| Average selected fine-tuned weights rather than selecting the best individual model | T5 · CoLA | Matthews correlation: +0.4 | Helped |
| Use high EMA weights instead of low EMA weights for greedy souping (ablation) | ViT-G/14 fine-tuned on ImageNet · ImageNet | validation accuracy | Hurt |
| Average models fine-tuned on different datasets with the zero-shot model | CLIP backbone · CIFAR-100 zero-shot evaluation | accuracy: +6.4 percentage points | Helped |
Closest papers by meaning
- Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging (ICLR 2024 · similarity 0.88)
- LT-Soups: Bridging Head and Tail Classes via Subsampled Model Soups (NeurIPS 2025 · similarity 0.87)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.87)
- Robust Fine-tuning of Zero-shot Models via Variance Reduction (NeurIPS 2024 · similarity 0.87)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.87)
- Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.