Research map / Large language models / Instruction and fine-tuning
Group-wise Data Ordering: Enhancing Instruction Tuning of Large Language Models via Embedding Proximity
Yiwen Ye, Boyuan Jiang, Xiaobin Hu, Shengzhi Wang, Xiaozhong Ji, Jinghao Lin, Deli Yu, Jiale Chen, Kai WU, Haihua Yang, Yong Xia
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| order clusters from easy to hard using embedding-proximity grouping with mixed regions and replay buffer, versus random shuffling | LLaVA-Vicuna-7B full fine-tuning · 12 multimodal sub-benchmarks avg | average score: 64.88 -> 66.30 | Improved |
| group-based ordering alone, without mixed regions (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 64.88 -> 64.70 | No clear effect |
| add mixed regions between clusters (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 64.70 -> 65.53 | Improved |
| add replay buffer data from earlier clusters to mixed regions (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 65.53 -> 66.30 | Improved |
| use embedding-based grouping instead of random partitions with the same mixed region and replay buffer design (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 64.59 -> 66.30 | Improved |
| use HDBSCAN (15 clusters) instead of k-means (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 65.53 -> 66.30 (k=15 vs HDBSCAN) | Improved |
| use too few (k=5) k-means clusters (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 65.64 (k=10) -> 64.14 (k=5) | Hurt |
| fold-wise cluster ordering instead of global easy-to-hard (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 66.30 -> 64.90 (n=5) | Hurt |
| order clusters hard-to-easy instead of easy-to-hard (ablation) | EP-Order · 12 multimodal sub-benchmarks avg | average score: 66.30 -> 65.60 | Hurt |
| sort data by loss from the warm-up model instead of random shuffle | LLaVA-Vicuna-7B full fine-tuning · 12 multimodal sub-benchmarks avg | average score: 64.88 -> 52.58 | Hurt |
| EP-Order versus random shuffling on LoRA fine-tuning | LLaVA-Vicuna-7B LoRA · 12 multimodal sub-benchmarks avg | average score: 64.38 -> 65.76 | Improved |
| EP-Order versus random shuffling on text-only instruction tuning | Qwen3-4B on Tulu-3-SFT · ARC, BBH, GSM, MMLU avg | average accuracy: 74.81 -> 79.51 | Improved |
Closest papers by meaning
- Steering Information Utility in Key-Value Memory for Language Model Post-Training (NeurIPS 2025 · similarity 0.88)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- CoIN: A Benchmark of Continual Instruction Tuning for Multimodel Large Language Models (NeurIPS 2024 · similarity 0.88)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.87)
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.87)
- Focus On This, Not That! Steering LLMs with Adaptive Feature Specification (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
