Research map / Large language models / Instruction and fine-tuning
Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMS-FTP, a partition-based multi-stage PEFT framework that groups domains by synergy and discrepancy into stages, with generalization bounds; reports gains over FULL, DA, PEFT and data-selection baselines on NSum, Q&A, Sentiment and Topic with LLaMA2 and Falcon.
Problem
Adapting a single LLM to multiple heterogeneous domains suffers from inter-domain interference, and existing multi-domain and PEFT methods do not exploit synergistic domain relationships or decide how to partition domains.
Approach
Partition source domains into sequential stages by maximizing an objective that rewards within-stage synergy (vocabulary Jaccard plus embedding cosine) and penalizes within-stage JS discrepancy and capacity cost, then fine-tune stage-wise with norm-bounded backbone and adapter updates.
Key results
- PMS-FTP reaches 44.2 NSum, 69.1 Q&A, 91.1 Sent, 89.0 Topic on Falcon-40B, above the best listed baseline (S2L at 43.5/67.8/90.7/88.4).
- On LLaMA2-7B, PMS-FTP peak GPU memory is 18.4 GB versus 27.2 GB for full fine-tuning (about 32% lower), slightly above LLaMA-Adapter's 17.2 GB.
- Ablation on NSum and Q&A with LLaMA2-7B: synergy-based two-stage partition (42.2 / 64.8) beats random two-stage (41.7 / 63.9) and single-stage (41.2 / 63.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random grouping with synergy-driven domain grouping (ablation) | PMS-FTP, LLaMA2-7B · NSum and Q&A | ROUGE-L and EM: 41.7 -> 42.2; 63.9 -> 64.8 | Helped |
| use one stage for all domains instead of two synergy-based stages (ablation) | PMS-FTP, LLaMA2-7B · NSum and Q&A | ROUGE-L and EM: 42.2 -> 41.2; 64.8 -> 63.1 | Hurt |
| use one domain per stage instead of two synergy-based stages (ablation) | PMS-FTP, LLaMA2-7B · NSum and Q&A | ROUGE-L and EM: 42.2 -> 42.0; 64.8 -> 64.2 | Hurt |
| set update norm bounds to (0.05, 0.05) rather than (0.1, 0.1) (ablation) | PMS-FTP, LLaMA2-7B · NSum and Q&A | ROUGE-L and EM: 42.2 -> 41.2; 64.9 -> 63.9 | Hurt |
| replace joint JS, vocabulary, and embedding partition signals with JS divergence only (ablation) | PMS-FTP, LLaMA2-7B, 4-domain slice · Q&A and NSum | F1 and ROUGE-L: 70.5 -> 69.3; 40.9 -> 40.0 | Hurt |
| replace joint JS, vocabulary, and embedding partition signals with embedding cosine only (ablation) | PMS-FTP, LLaMA2-7B, 4-domain slice · Q&A and NSum | F1 and ROUGE-L: 70.5 -> 69.6; 40.9 -> 40.3 | Hurt |
| ignore synergy by setting λ=0 (ablation) | PMS-FTP, LLaMA2-7B, 4-domain slice · Q&A and NSum | F1 and ROUGE-L: 70.5 -> 70.3; 40.9 -> 40.8 | No clear effect |
| use synergy-only weighting (λ=1.0) (ablation) | PMS-FTP, LLaMA2-7B, 4-domain slice · Q&A and NSum | F1 and ROUGE-L: 70.5 -> 70.2; 40.9 -> 40.6 | No clear effect |
| reverse the stage order (ablation) | PMS-FTP, LLaMA2-7B, 4-domain slice · Q&A and NSum | F1 and ROUGE-L: 70.5 -> 70.4; 40.9 -> 40.8 | No clear effect |
| interleave training between the two stages (ablation) | PMS-FTP, LLaMA2-7B, 4-domain slice · Q&A and NSum | F1 and ROUGE-L: 70.5 -> 70.3; 40.9 -> 40.7 | No clear effect |
| replace a single domain centroid with K=3 weighted barycenters (ablation) | PMS-FTP, LLaMA2-7B · NSum, Q&A, Sent, Topic | ROUGE-L, EM, and accuracy: 43.4 -> 43.3; 67.2 -> 67.1; 90.2 -> 90.1; 88.0 -> 87.9 | No clear effect |
Surprising findings
- Stage ordering changes results by at most 0.2 points, which the authors read as negligible sensitivity.
- Domain adaptation baselines (MDAN, M3SDA, GMDI) underperform plain full fine-tuning across all tasks in the main table.
Limitations
- Theoretical bounds rely on strong assumptions (Lipschitz outputs, norm-bounded parameters) and a proof sketch whose Rademacher step is loose; the bound term is not shown to be tight.
- Experiments cover only four domains in the main table with small ablations; code and data are not released.
- Several reported results (e.g., many appendix tables, bootstrap CIs, 12-domain experiments) lack clear protocol details and show internal inconsistencies between the main paper and appendix.
Experimental setup
Datasets: News Summarization (NSum) Sentiment Classification (Sent) Question Answering (Q&A) Topic Categorization (Topic) SQuAD HotpotQA CNN/DM XSum Wiki-10 Multi-News
Benchmarks: NSum Q&A Sentiment Topic HellaSwag MMLU-STEM ARC-easy SciQ GSM8K
Metrics: ROUGE-L exact match F1 accuracy pass@1 peak GPU memory
Baselines: FULL FIXED MDAN M3SDA GMDI LoRA Adapter LLaMA-Adapter Q-LoRA Tag-LLM INSTRUCTMINING S2L
Builds on: LoRA domain-adversarial training (MDAN) M3SDA S2L InstructMining
Closest papers by meaning
- Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples (ICML 2025 · similarity 0.90)
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning (ICLR 2024 · similarity 0.89)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.89)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.89)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.88)
- FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.