Research map / Large language models / Instruction and fine-tuning

FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Instruction data selection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Selects a fixed-budget subset of SFT training sentences greedily by maximizing the log-determinant of a last-layer Fisher information approximation, with a fast batched/lazy-greedy implementation, and evaluates on synthetic data, word2vec embeddings, and GPT-2 fine-tuning on tiny-Shakespeare.

Problem

Supervised fine-tuning of LLMs is costly, so choosing which n training sentences to fine-tune on should be done efficiently and in a way that maximizes statistical information for the fine-tuned model.

Approach

Views the LLM's last softmax layer as a product of multinomial logistic regressions, bounds the Hessian log-determinant by a d×d design matrix built from pre-logit embeddings (removing dependence on vocabulary size L and on the unknown parameters), and greedily selects sentences maximizing log det, using lazy greedy with submodularity.

Key results

• On synthetic tokens, FisherSFT reaches the best baseline's lowest maximum error (at n=2000) with only n=1000 sentences.
• On GPT-2 fine-tuning on tiny-Shakespeare, GPT-4o judge prefers FisherSFT over uniform sampling in 0.56–0.80 of comparisons across sample sizes 100–5000.
• Against density sampling, FisherSFT is preferred in 0.54–0.84 of comparisons; against ASK-LLM, 0.52–0.74.

Measured effects reported in the paper

Limitations

• Theory relies on the last-layer linearization and assumptions (bounded features, diversity kappa), with the error bound applying only to the linearized multinomial logistic model.
• GPT-2 evaluation uses a single small corpus and an LLM judge with a position bias of 0.54, with no human evaluation.
• Fine-tuning evaluation is qualitative (preference win rates) rather than measuring downstream task accuracy.

Experimental setup

Datasets: tiny-Shakespeare word2vec pre-trained embeddings (synthetic token setup)

Benchmarks: synthetic autoregressive prediction word2vec token prediction tiny-Shakespeare generation quality

Metrics: maximum prediction error mean prediction error win fraction vs baseline

Baselines: uniform sampling sentence-level Fisher (log det) density sampling clustering-based sensitivity sampling ASK-LLM

Builds on: Fisher information / optimal experimental design ASK-LLM density sampling

Closest papers by meaning

• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.93)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models (AAAI 2025 · similarity 0.90)
• Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.90)
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Fine-Tuning Language Models with Collaborative and Semantic Experts (AAAI 2025 · similarity 0.89)
• DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.89)