Research map / Large language models / Instruction and fine-tuning

Preference-Oriented Supervised Fine-Tuning: Favoring Target Model over Aligned Large Language Models

AAAI 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PoFT, an SFT objective that applies a Bradley-Terry preference loss favoring the target model's length-normalized likelihood over that of aligned LLMs on the same SFT data, improving Mistral-7B and Llama-3-8B over cross-entropy SFT.

Problem

Standard cross-entropy SFT is sensitive to the quality of instruction-response data, and high-quality SFT datasets are costly to create and maintain.

Approach

Treat SFT as a Bradley-Terry preference between the target model and aligned LLMs on the same (x,y) data, using length-normalized log-likelihoods as rewards, so the aligned LLMs' likelihoods act as data-quality signals that reweight per-sample gradients.

Key results

• On the Open LLM Leaderboard with OpenHermes, PoFT improved Mistral-7B overall score from 61.94 to 63.52 and Llama-3-8B by 2.17 points (63.06 to 65.23).
• With UltraChat200k, PoFT gave small gains (e.g., Mistral-7B 63.26 to 63.74 overall), and on MT-Bench/AlpacaEval the OpenHermes-trained Mistral-7B rose from 5.09 to 5.93 and win rate from 4.86% to 5.96%.
• PoFT followed by DPO on Mistral-7B raised AlpacaEval 2.0 win rate from 25.09% (SFT+DPO) to 27.83%.

Measured effects reported in the paper

Surprising findings

• Directly regenerating OpenHermes responses with aligned LLMs (e.g., Yi-6B-Chat) often degraded SFT performance, in some cases below the original-data baseline.
• Combining PoFT with IFD, Instag, or Deita filtering yields further gains over either alone, though filtering alone already surpasses PoFT on full data.

Limitations

• Gains are small on some datasets (e.g., UltraChat200k gains under 0.5 points) and MT-Bench/AlpacaEval differences are modest.
• Relies on the assumption that aligned LLMs provide valid data-quality assessments; the choice of aligned LLMs affects results inconsistently.
• Evaluation uses a single seed-free comparison with std reported only across epochs; no significance testing.

Experimental setup

Datasets: UltraChat200k OpenHermes ShareGPT-Chinese-English-90k UltraFeedback Alpaca Dolly

Benchmarks: HuggingFace Open LLM Leaderboard (ARC, TruthfulQA, Winogrande, GSM8k, HellaSwag, MMLU) MT-Bench AlpacaEval 2.0

Metrics: accuracy normalized accuracy MT-Bench score AlpacaEval win rate standard deviation across epochs

Baselines: cross-entropy SFT Zephyr-7B-sft-full IFD Instag Deita DPO

Builds on: Direct Preference Optimization Bradley-Terry model Zephyr-7B-sft-full alignment-handbook IFD

Closest papers by meaning

• Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data (ICML 2025 · similarity 0.93)
• FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain (ICML 2025 · similarity 0.90)
• From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning (ICML 2024 · similarity 0.90)
• Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.90)
• InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)