Research map / Large language models / Instruction and fine-tuning

One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DualSFT, a one-shot dual-scoring method that derives parameter masks and data subsets for LLM fine-tuning from one shared gradient interaction matrix (row/column aggregations), with CWSD for preserving prior capabilities.

Problem

Parameter selection and data selection for cost-efficient LLM fine-tuning are scored separately, causing redundant overhead and coordination mismatch when both restrict the update jointly.

Approach

Casts parameter and data selection as bilevel problems under a shared validation objective; under first/second-order local surrogates, parameter importance and data utility are column-wise and row-wise sums of one gradient interaction matrix, so both masks come from one projection vector u computed from shared gradient statistics, plus a pretraining-data-free self-distillation term (CWSD) for rete

Key results

• On Llama-3.2-3B with Magicoder, DualSFT reaches 43.67% HumanEval under joint budget, 4.15 points above SPICE+NanoAdam.
• Full DualSFT Overall of 47.10% on Llama-3.2-3B, 56.50% on Gemma-3-4B-PT, 72.95% on Qwen-3.5-9B-Base (Magicoder), outperforming strongest sequential hybrid by 2.39/2.61/3.24 points.
• DualSFT-Param reports 19.3 GB peak fine-tuning memory; full DualSFT 28.5 GB peak selection memory and 7.4 GPU-hours.

Surprising findings

• Re-selection with fixed-side passes improves Overall by at most 0.12 while increasing selection and end-to-end cost by 5-45%, suggesting one-shot scoring captures most of the joint structure.
• Reverse matched-budget selections underperform random controls, indicating score directionality matters beyond magnitude or sparsity.

Limitations

• Theory relies on local one-step/second-order surrogates and diagonal curvature approximation; no exact bilevel optimum recovered.
• Evaluated only on code and math SFT with 3B-9B models.
• Parameter scoring requires O(D) auxiliary state and full-model vectors.

Experimental setup

Datasets: Magicoder MetaMathQA HumanEval PIQA ARC-C MMLU GSM8K

Benchmarks: HumanEval PIQA ARC-C MMLU GSM8K

Metrics: accuracy plasticity stability overall score

Baselines: Standard SFT LoRA S^2FT SMT LIFT NanoAdam Random LESS FisherSFT SPICE LESS + S^2FT SPICE + NanoAdam

Builds on: In-Run Data Shapley LESS S^2FT Ghost Dot Product LlamaFactory

Closest papers by meaning

• LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.91)
• Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs (ICLR 2024 · similarity 0.90)
• BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training (NeurIPS 2026 · similarity 0.90)
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
• Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.90)
• When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)