Research map / Architectures and efficiency / Pruning and efficient tuning
Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging
MINSIK CHOI, Geewook Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace centralized joint training (1 ep) with PCA conflict-aware 3D split and one-shot token-weighted merge | MERIT · 8-benchmark average, Vision-FLAN | average score: 54.3 -> 57.0 | Improved |
| use conflict-aware PCA split instead of random partitioning (8 groups) (ablation) | MERIT · 8-benchmark average, Vision-FLAN | average score: 54.5 -> 57.0 | Improved |
| increase number of PCA dimensions from 1D to 2D to 3D (ablation) | MERIT · 8-benchmark average, Vision-FLAN | average score: 55.2 -> 55.7 -> 57.0 | Improved |
| increase number of random groups from 2 to 8 (ablation) | random partition and merge · 8-benchmark average, Vision-FLAN | average score: 54.6 -> 54.4 -> 54.5 | No clear effect |
| even random partitioning plus merge versus joint 1 ep (ablation) | split and merge · 8-benchmark average, Vision-FLAN | average score: 54.3 -> 54.6 | Improved |
| use PCA split instead of greedy conflict-induced split (2 groups) (ablation) | MERIT 1D · 8-benchmark average, Vision-FLAN | average score: 54.9 -> 55.2 | Improved |
| use uniform soup of 2 models on full mixture instead of 1D MERIT (ablation) | soup baseline · 8-benchmark average, Vision-FLAN | average score: 55.4 vs 55.2 | Mixed |
| train 7B from LLaVA-style base with MERIT 2D instead of joint FFT | Base VLM 7B · 8-benchmark average, 1.6M mixture | average score: 54.9 -> 55.4 | Improved |
| MERIT 2D instead of joint FFT on stronger 3.6M base | Scaled base VLM 7B · 8-benchmark average, 1.6M mixture | average score: 60.9 -> 61.5 | Improved |
| use MERIT 2D instead of joint FFT on stronger base | Scaled base VLM 7B · LLaVA-W | score: 50.2 -> 66.2 | Improved |
| MERIT 2D instead of joint FFT on stronger base | Scaled base VLM 7B · MathVista | score: 49.7 -> 43.0 | Hurt |
| MERIT 2D instead of 1-epoch joint training on text-only FLAN | Qwen2.5-3B · Text-only FLAN, 8-benchmark average | average score: 57.6 -> 58.4 | Improved |
Closest papers by meaning
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- DISTMOE: Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning (NeurIPS 2026 · similarity 0.87)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.87)
- Pilot: Building the Federated Multimodal Instruction Tuning Framework (AAAI 2025 · similarity 0.87)
- Mixture-Trained Merging for Unified Multi-Objective Models (NeurIPS 2026 · similarity 0.87)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
