Research map / Architectures and efficiency / Pruning and efficient tuning
GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry
Guanghui Min, Tianhao Huang, Ke Wan, Chen Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GIST subspace-alignment selection instead of LESS | Llama3.2-3B · TYDIQA | accuracy: 67.1 -> 69.2 | Improved |
| use GIST instead of LESS | Llama2-7B · MMLU, TYDIQA, BBH | accuracy: LESS 50.2/56.2/41.5 vs GIST 51.2/55.8/41.8 | Mixed |
| use GIST instead of LESS | Qwen2.5-1.5B · BBH | accuracy: 43.2 -> 44.1 | Improved |
| select 5% data with GIST vs full-data fine-tuning | Llama3.2-3B · Avg. delta over Base | accuracy: +3.3 -> +4.5 | Improved |
| add GIST selection (average over three benchmarks) | Qwen2.5-1.5B · Avg. delta over Base | accuracy: +1.1 (LESS) -> +1.5 | Improved |
| perplexity-based selection instead of random | Llama2-7B · Avg. delta over Base | accuracy: -8.0 | Hurt |
| use only one early checkpoint (epoch 1) instead of 4 (ablation) | GIST on Llama3.2-3B · TYDIQA | accuracy: 64.3 -> 69.2 | Improved |
| aggregate all four checkpoints (ablation) | GIST on Llama3.2-3B · TYDIQA | accuracy: 69.2 -> 67.3 | Hurt |
| aggregate epochs 1-2 (ablation) | GIST on Llama3.2-3B · TYDIQA | accuracy: 69.2 -> 70.2 | Improved |
| use full rank 285 instead of rank 150 subspace (ablation) | GIST on MMLU · MMLU | accuracy: 62.9 -> 62.5 | Hurt |
| rank 10 subspace (40% explained variance) instead of 150 (ablation) | GIST on MMLU · MMLU | accuracy: 62.9 -> 62.1 | No clear effect |
| single-epoch GIST vs LESS 4-epoch compute | GIST vs LESS · runtime and storage | A100 hours: LESS 6.0 h + 48.0 h vs GIST 1.5 h + 12.0 h | Improved |
Closest papers by meaning
- Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations (NeurIPS 2025 · similarity 0.86)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.85)
- Online Data Selection for Instruction Tuning via Gaussian Processes (NeurIPS 2026 · similarity 0.85)
- LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection (NeurIPS 2026 · similarity 0.84)
- GIO: Gradient Information Optimization for Training Dataset Selection (ICLR 2024 · similarity 0.84)
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
