Research map / Large language models / Instruction and fine-tuning
Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning
Ling Zhang, Xianliang Yang, Juwon Yu, Park Cheonyoung, Lei Song, Jiang Bian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reweight SFT examples by ICA score | SFT LLaMA-3-8B · Yahoo_Answers_Topics | win rate vs no reweighting: 85.10% | Improved |
| reweight DPO examples by ICA score | DPO LLaMA-3-8B · SHP-2 | win rate vs no reweighting: 77.20% | Improved |
| ICA instead of RHO-Loss (no reference model) | reweighting · SFT/DPO/SimPO | win rate vs RHO-Loss: about 46-56% | No clear effect |
| ICA instead of one-shot learning scoring | reweighting · all settings | win rate vs One-Shot: 49.60-66.93% | Improved |
| use top-k=1 or 10 holdout demonstrations instead of 3 (ablation) | ICA reweighting (LLaMA-3-3B) · Yahoo_Answers_Topics | win rate vs default: 43.50% (k=1), 46.07% (k=10) | No clear effect |
| increase score updates R from 1 to 5 (ablation) | ICA reweighting (LLaMA-3-3B) · Yahoo_Answers_Topics | win rate vs default: 52.77% | Improved |
| increase R to 9 (ablation) | ICA reweighting (LLaMA-3-3B) · Yahoo_Answers_Topics | win rate vs default: 51.60% | No clear effect |
| hard percentile filtering instead of reweighting (ablation) | ICA scores (LLaMA-3-3B) · Yahoo_Answers_Topics | win rate vs reweighting: 40.80% (p50), 48.67% (p75), 40.07% (p90) | Hurt |
| use stronger bge-m3 embeddings for kNN (ablation) | ICA reweighting (LLaMA-3-3B) · Yahoo_Answers_Topics | win rate vs default: 52.13% | Improved |
| ICA reweighting on corrupted GSM8K CoT | SFT LLaMA-3-8B · GSM8K (40% corrupted) | Pass@1: 52.75 -> 79.95 | Improved |
| ICA vs RHO-Loss on LLaMA-3-8B GSM8K corrupted | reweighting · GSM8K (40% corrupted) | Pass@1: 79.50 -> 79.95 | No clear effect |
| ICA vs LESS / GREATS selection | SFT LLaMA-3-8B · Yahoo_Answers_Topics | win rate: 52.03% vs GREATS, 51.40% vs LESS | No clear effect |
Closest papers by meaning
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.89)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.89)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.89)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
