Research map / Large language models / Instruction and fine-tuning

Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Instruction data selection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes T-SPIN, a triplet-based self-play fine-tuning method that adds historical advantages against proto-synthetic responses from the initial policy and an entropy-constrained, reference-free reward, improving stability over SPIN and matching SFT with 25% of the data.

Problem

Self-play fine-tuning (SPIN) for LLMs with scarce annotated data suffers unstable optimization, as the current reward advantage vanishes during iterations, and a misalignment between the reference-based training reward and the log-likelihood used for generation.

Approach

Augment the main player's objective with a historical advantage between synthetic responses and proto-synthetic responses from the initial policy, and derive via an entropy-constrained opponent objective a reference-free reward (alpha times log-likelihood) that aligns training with generation.

Key results

• On Zephyr-7B, T-SPIN reaches 43.47% average at iteration 4 versus 40.62% for SPIN and 42.01% for SFT trained on 200k samples.
• T-SPIN improves GSM8K by 14.82 points and IFEval by 28.32 points over Zephyr-7B base.
• On Mistral-7B, T-SPIN iteration 4 averages 45.02% versus 42.32% for SPIN and 44.17% for SFT.

Measured effects reported in the paper

Surprising findings

• SPIN's log-likelihood of annotated responses falls below that of synthetic responses in later iterations despite higher training reward, showing reward-generation misalignment.
• Removing the historical advantage (HA variant) causes performance drops at early iterations (e.g., GSM8K falls at iteration 1).

Limitations

• Synthetic responses must be regenerated for all prompts each iteration, with no filtering of samples that no longer contribute.
• The annotated data is fixed, so the method does not address non-stationary target distributions.

Experimental setup

Datasets: UltraChat200k

Benchmarks: GSM8K MATH MUSR MMLU MMLU-Pro GPQA HellaSwag WinoGrande BBH IFEval

Metrics: accuracy normalized accuracy average score across 10 tasks

Baselines: SPIN SFT Zephyr-7B Mistral-7B HA (no historical advantage) TR (triplet with reference reward)

Builds on: SPIN direct preference optimization integral probability metric

Closest papers by meaning

• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.94)
• IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning (NeurIPS 2026 · similarity 0.91)
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.89)
• One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuning (NeurIPS 2026 · similarity 0.88)
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.88)