Research map / Large language models / Instruction and fine-tuning

Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 13 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DARE, a training-free method that randomly drops most SFT delta parameters and rescales the rest, showing they are highly redundant and that the sparsified models can be merged with existing merging methods to combine abilities.

Problem

SFT delta parameters (fine-tuned minus pre-trained weights) appear highly redundant, and merging multiple homologous SFT models suffers from parameter interference; the paper asks whether models can absorb abilities from homologous models without retraining or GPUs.

Approach

Randomly drop a large fraction (e.g., 90-99%) of delta parameters and rescale remaining ones by 1/(1-p) so expected embeddings are preserved; use this as a plug-in sparsifier before merging SFT models (Task Arithmetic, TIES-Merging, etc.) to reduce interference.

Key results

• DARE can remove 90% or even 99% of SFT delta parameters with little loss; WizardMath-70B tolerates p=0.99 while 7B and 13B variants fail.
• Merging WizardLM-13B, WizardMath-13B and llama-2-13b-code-alpaca with Task Arithmetic plus DARE improves over the source LM by 3.10% on AlpacaEval and over Math by 3.18% on GSM8K, and over Code by 19.57% on MBPP.
• A 7B merged model supermario_v2 reached first rank on the Open LLM Leaderboard (average 75.49) as of January 28, 2024, using CPU-only merging.

Measured effects reported in the paper

Limitations

• DARE works only when SFT delta parameters are small (roughly within 0.002); it fails for continually pre-trained or mismatched backbones.
• Merged models do not always surpass single source models on encoder-based GLUE tasks; gains are modest (about 0.3-0.8% average).
• Drop rate tolerance must be tuned per model; no theory for the upper bound of p.

Experimental setup

Datasets: GLUE AlpacaEval GSM8K MATH HumanEval MBPP Open LLM Leaderboard tasks

Benchmarks: AlpacaEval GSM8K MATH HumanEval MBPP GLUE Open LLM Leaderboard

Metrics: win rate zero-shot accuracy pass@1 Matthews correlation accuracy F1 Pearson correlation Spearman correlation cosine similarity of embeddings

Baselines: Average Merging Task Arithmetic Fisher Merging RegMean TIES-Merging Magnitude-based Pruning DropOnly Multi-task learning

Builds on: Task Arithmetic TIES-Merging Dropout magnitude-based pruning

Closest papers by meaning

• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
• Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models (NeurIPS 2024 · similarity 0.89)
• Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.89)
• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.88)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.88)
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.88)