Research map / Large language models / Instruction and fine-tuning
STAFF: Speculative Coreset Selection for Task-Specific Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAFF, a speculative coreset selection method that scores data with a small same-family LLM and verifies region importance on the target LLM to allocate selection budget for task-specific fine-tuning.
Problem
Coreset selection for task-specific LLM fine-tuning degrades at high pruning rates when important or representative samples in low-score regions are dropped, and existing methods are very costly because they fine-tune the target LLM to score data.
Approach
Use a small model from the same family (e.g., Gemma-2b for Gemma-7b) to fine-tune and compute effort-score importance cheaply, then verify region-level scores on the target LLM and reallocate the selection budget toward regions the target finds important via stratified sampling.
Key results
- STAFF improves over the best baseline by up to 54.3% in fine-tuning performance and reduces selection overhead by up to 70.5% across pruning rates 20%-90%.
- On DialogSum at 20% pruning, STAFF gives an average improvement of 0.2 over full-dataset fine-tuning.
- Replacing the same-family small model with LLaMA-160M causes the largest ablation drop, with results near random selection at many pruning rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STAFF coreset selection instead of CCS | Gemma-7b fine-tuning · BioInstruct, 20% pruning | Rouge-L: 41.7 -> 42.0 | Helped |
| Use STAFF coreset selection instead of CCS | Gemma-7b fine-tuning · DialogSum, 90% pruning | Rouge-L: 47.4 -> 48.4 | Helped |
| Use STAFF coreset selection instead of CCS | Gemma-7b fine-tuning · WMT-19, 90% pruning | Rouge-L: 54.1 -> 55.9 | Helped |
| Use STAFF coreset selection instead of GraNd | Gemma-7b coreset selection · BioInstruct, 90% pruning | selection time (h): 5.1 -> 3.1 | Helped |
| Use STAFF coreset selection instead of GraNd | Gemma-7b coreset selection · DialogSum, 90% pruning | selection time (h): 4.4 -> 3.0 | Helped |
| Use STAFF coreset selection instead of GraNd | Gemma-7b coreset selection · WMT-19, 90% pruning | selection time (h): 8.7 -> 3.5 | Helped |
| Add target-LLM verification and verification-based budget allocation (ablation) | STAFF on Gemma-7b · WMT-19, 90% pruning | Rouge-L: 54.0 -> 55.9 | Helped |
| Add target-LLM verification and verification-based budget allocation (ablation) | STAFF on Gemma-7b · BioInstruct, 20% pruning | Rouge-L: 41.9 -> 42.0 | No clear effect |
| Add speculative scoring with a same-family small model (ablation) | STAFF on Gemma-7b · BioInstruct, 20% pruning | Rouge-L: 41.0 -> 42.0 | Helped |
| Add speculative scoring with a same-family small model (ablation) | STAFF on Gemma-7b · WMT-19, 90% pruning | Rouge-L: 55.4 -> 55.9 | No clear effect |
| Use a same-family small model rather than LLaMA-160M for speculative scores (ablation) | STAFF on Gemma-7b · BioInstruct, 20% pruning | Rouge-L: 40.8 -> 42.0 | Helped |
| Use a same-family small model rather than LLaMA-160M for speculative scores (ablation) | STAFF on Gemma-7b · WMT-19, 90% pruning | Rouge-L: 54.8 -> 55.9 | Helped |
Surprising findings
- A 20% coreset selected by STAFF on DialogSum slightly outperforms fine-tuning on the full dataset.
- Using a small model from a different family (LLaMA-160M) degrades STAFF to near-random selection performance.
Limitations
- Relies on a same-family small model being available; cross-family small models performed poorly.
- Evaluated on only three LLMs and three tasks with a single fine-tuning setup (LoRA, 4 epochs).
Experimental setup
Datasets: BioInstruct DialogSum WMT-19 Kazakh-English
Benchmarks: BioInstruct DialogSum WMT-19 Kazakh-English
Metrics: Rouge-L BLEU-4 selection time overhead (h)
Baselines: Random GraNd EL2N CCS D2 Pruning
Builds on: GraNd EL2N CCS D2 Pruning LoRA speculative decoding
Closest papers by meaning
- Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.88)
- Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.88)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.87)
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models (NeurIPS 2024 · similarity 0.87)
- LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.87)
- One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.