Research map / Large language models / Instruction and fine-tuning
Not All Documents Are What You Need for Extracting Instruction Tuning Data
Chi Zhang, Huaping Zhong, Hongtao Li, Chengliang Chai, Hongjiawei, Yu-Ping Wang, Yuhao Deng, Jiacheng Wang, Yizhou Yan, Qiu Jiantao, Conghui He, Lei Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive-learning warm-up for document embeddings (ablation) | EQUAL · GSM8K (Full FT, 5% docs) | accuracy: 69.73 -> 73.01 | Improved |
| add multi-armed bandit iterative selection (ablation) | EQUAL · GSM8K (Full FT, 5% docs) | accuracy: 71.90 -> 73.01 | Improved |
| use optimal transport reward instead of average similarity (ablation) | EQUAL · GSM8K (Full FT, 5% docs) | accuracy: 70.77 -> 73.01 | Improved |
| add contrastive-learning warm-up (ablation) | EQUAL · HumanEval (Full FT, 5% docs) | pass rate: 44.3 -> 49.4 | Improved |
| add multi-armed bandit iterative selection (ablation) | EQUAL · HumanEval (Full FT, 5% docs) | pass rate: 46.9 -> 49.4 | Improved |
| use optimal transport reward (ablation) | EQUAL · HumanEval (Full FT, 5% docs) | pass rate: 47.6 -> 49.4 | Improved |
| add contrastive-learning warm-up (ablation) | EQUAL · GSM8K (LoRA, 5% docs) | accuracy: 64.05 -> 67.32 | Improved |
| use 1000 clusters instead of k=50 (ablation) | EQUAL · GSM8K and MATH | accuracy: 2.90% and 2.05% lower accuracy with k=50 | Improved |
| use 1000 clusters instead of k=50,000 (ablation) | EQUAL · GSM8K and MATH | accuracy: 2.56% and 1.79% lower accuracy with k=50,000 | Improved |
| swap k-means for BIRCH or DBSCAN clustering (ablation) | EQUAL · downstream tasks | accuracy | No clear effect |
| raise extracted document ratio from 5% to 20% | EQUAL · MATH (Full FT) | accuracy: 35.10 -> 41.40 | Improved |
| use EQUAL instead of Influence selection | Llama-3-8B Full FT · GSM8K (Full FT, 5% docs) | accuracy: 65.20 -> 73.01 | Improved |
Closest papers by meaning
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.89)
- Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning (ICLR 2025 · similarity 0.87)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.87)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.87)
- T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuning (NeurIPS 2025 · similarity 0.87)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
