Research map / Large language models / Instruction and fine-tuning
Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EVIC, a curriculum that periodically recomputes inter-sample influence (via JL-projected Adam gradients) and trains only on samples with non-negative total influence, improving multi-domain (code, math, general) LLM fine-tuning and performance-to-sample ratio.
Problem
Multi-domain LLM fine-tuning suffers a trade-off across abilities (code, math, general) because samples conflict; prior empirical or heuristic data management yields marginal gains and lacks a principled model of inter-sample interactions.
Approach
Model inter-sample interactions as the influence of one sample's (Adam) gradient on another's loss, observe these interactions evolve during training, and iteratively select samples with non-negative total influence on the dataset for training.
Key results
- EVIC reaches average 37.4 on Mistral-7B vs 33.3 for the best baseline (MoS), roughly four points higher.
- EVIC achieves performance-to-sample ratio at least 1.29x that of MTL across the three models (2.11x on Qwen2.5-14B).
- Nearly 90% of samples are learned within two iterations and over 96% within three iterations on Mistral-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EVIC instead of the strongest baseline, MoS | Mistral-7B-v0.3 multi-domain fine-tuning · Multi-domain benchmark average | AVG: 33.3 -> 37.4 | Helped |
| Use EVIC instead of the strongest baseline, DMT | Llama-3.1-8B multi-domain fine-tuning · Multi-domain benchmark average | AVG: 43.5 -> 44.3 | Helped |
| Use EVIC instead of the strongest baseline, MoS | Qwen2.5-14B multi-domain fine-tuning · Multi-domain benchmark average | AVG: 56.0 -> 57.0 | Helped |
| Use EVIC instead of MTL | Mistral-7B-v0.3 multi-domain fine-tuning · GSM8K-test | accuracy: 56.2 -> 57.1 | Helped |
| Use EVIC instead of MoS | Mistral-7B-v0.3 multi-domain fine-tuning · HumanEval | Pass@1: 28.6±0.7 -> 37.8±0.5 | Helped |
| Use EVIC instead of DMT | Llama-3.1-8B multi-domain fine-tuning · AlpacaEval 2.0 | LC-WR: 12.9±1.0 -> 11.6±0.6 | Hurt |
| Use EVIC instead of MoS | Qwen2.5-14B multi-domain fine-tuning · HumanEval | Pass@1: 66.7±0.9 -> 69.1±0.9 | Helped |
| Use EVIC instead of DMT | Qwen2.5-14B multi-domain fine-tuning · AlpacaEval 2.0 | LC-WR: 18.1±0.9 -> 17.9±0.9 | Hurt |
| Iteratively update the interaction matrix and resample training examples rather than using a fixed selection after warm-up (ablation) | EVIC on Mistral-7B-v0.3 · Multi-domain benchmark average | AVG: 33.1 -> 37.4 | Helped |
| Iteratively update the interaction matrix and resample training examples rather than using a fixed selection after warm-up (ablation) | EVIC on Llama-3.1-8B · Multi-domain benchmark average | AVG: 43.2 -> 44.3 | Helped |
| Increase warm-up sample proportion from 5% to 20% or 100% (ablation) | EVIC on Mistral-7B-v0.3 · Multi-domain benchmark | Average performance: less than 2% | Mixed |
Surprising findings
- Sample interactions change sign and magnitude during training, so they are not fixed by inherent semantics.
- Interactions are asymmetric and some same-domain samples conflict while some cross-domain samples mutually promote each other.
Limitations
- Theoretical explanation of why interactions evolve and why nearly all samples are covered is left to future work.
- Evaluation limited to three domains and three benchmarks with LoRA fine-tuning; gains on Qwen2.5-14B are about one point over the best baseline.
Experimental setup
Datasets: CodeAlpaca GSM8K-RFT Alpaca-GPT4
Benchmarks: HumanEval GSM8K AlpacaEval 2.0
Metrics: Pass@1 Pass@10 accuracy length-controlled win rate win rate average score
Baselines: MTL DSL DMT MoS
Builds on: Learning dynamics influence estimation (Xia et al., 2024a) Mixed fine-tuning DMT (Dong et al., 2023) Mixture-of-Skills (Wu et al., 2024) Adam optimizer LoRA
Closest papers by meaning
- Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning (NeurIPS 2025 · similarity 0.90)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.90)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.89)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.89)
- EvoLM: In Search of Lost Language Model Training Dynamics (NeurIPS 2025 · similarity 0.89)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.