Research map / Large language models / Instruction and fine-tuning
Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
Bing Wang, Ximing Li, Changchun Li, Jinjin Chi, Gang Niu, Masashi Sugiyama
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BADIT (SVD-based ability decomposition plus orthogonal grouping) over GainLoRA | multi-task LoRA-MoE · SuperNI, mixed training | ROUGE: 54.33 -> 55.87 (Qwen3-8B) | Improved |
| add BADIT | LoRAMoE on Llama3-8B · SuperNI, sequential training | ROUGE: 45.47 -> 48.83 | Improved |
| add BADIT | GainLoRA on Qwen3-8B · SuperNI, sequential training | Forget Rate: 8.96 -> 6.95 | Improved |
| add SVD-based basic ability decomposition initialization (BAD) (ablation) | BADIT · SuperNI sequential, Qwen3-8B | ROUGE: 50.07 -> 50.86 | Improved |
| add dynamically orthogonal grouping (DOG) (ablation) | BADIT · SuperNI sequential, Qwen3-8B | ROUGE: 49.70 -> 50.86 | Improved |
| add both BAD and DOG (ablation) | LoRAMoE · SuperNI sequential, Llama3-8B | ROUGE: 45.47 -> 48.83 | Improved |
| add DOG (ablation) | BADIT · SuperNI mixed, Gemma2-9B | ROUGE: 56.03 -> 57.35 | Improved |
| add dynamic regrouping of rank-1 components | LoRAMoE on Qwen3-8B · gradient angles | inter-expert gradient angle: near 90 degrees maintained throughout training | Improved |
| add BADIT over LoRAMoE | LoRAMoE · training time | relative training time: 1x -> 1.22x | Hurt |
| add SVD-based initialization without DOG (ablation) | LoRAMoE · training time | relative training time: 1x -> 1.00x | No clear effect |
| use PiSSA instead of LoRA | multi-task LoRA · SuperNI mixed, Qwen3-8B | ROUGE: 54.22 -> 53.57 | Hurt |
Closest papers by meaning
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts (NeurIPS 2025 · similarity 0.87)
- Two-stage LLM Fine-tuning with Less Specialization and More Generalization (ICLR 2024 · similarity 0.87)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.87)
- How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.87)
- On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
