Research map / Large language models / Instruction and fine-tuning

SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning generalization and delta compression · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SPACE, a self-play fine-tuning method that replaces SPIN's relative-gap objective with an NCE-style binary classification loss that separately optimizes real and synthetic response rewards, yielding stable iterative gains on Open LLM Leaderboard tasks.

Problem

Self-play fine-tuning methods such as SPIN optimize the relative reward gap between real and self-generated responses; as the generator improves the gap vanishes and the objective degenerates to a constant, causing unstable evolution and performance decline over iterations.

Approach

Treat self-generated responses as auxiliary noise samples and train the main player as a binary classifier (noise contrastive estimation) that separately raises the absolute reward of real responses and lowers that of synthetic ones, so the objective never collapses to a constant; the opponent is then set to the updated main player.

Key results

• On Mistral-7B, SPACE raises the average Open LLM Leaderboard score from 48.42% to 52.43% at iteration 4, with GSM8K improving from 37.68% to 46.02% and IFEval from 23.63% to 35.90%.
• SPACE with 50k real responses is reported to outperform SFT trained on 200k samples, per the paper's comparison.
• SPIN peaks at iteration 2 (49.33 avg, then declines to 48.57 at iteration 4 on Mistral-7B), while SPACE improves monotonically across iterations.

Measured effects reported in the paper

Surprising findings

• In SPIN, the reward margin grows mainly because real-response rewards decline, which the authors view as undesirable; SPACE instead increases real-response rewards.
• Gap-based self-play variants (S-IPO, S-SimPO) and SPIN achieve their best averages in early iterations and degrade afterward.

Limitations

• Iteration count is a hyperparameter fixed in advance; adaptive stopping is left to future work.
• Method assumes a stationary target data distribution; non-stationary settings are not addressed.
• Gains on some benchmarks are modest (e.g., GPQA, BBH) and results are reported for only two 7B base models.

Experimental setup

Datasets: UltraChat200k

Benchmarks: ARC GSM8K HellaSwag MMLU TruthfulQA Winogrande IFEval BBH GPQA MMLU-Pro

Metrics: accuracy average score across tasks

Baselines: SPIN S-IPO S-SimPO SFT

Builds on: SPIN noise contrastive estimation IPO SimPO REINFORCE

Closest papers by meaning

• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.90)
• IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning (NeurIPS 2026 · similarity 0.90)
• Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs (NeurIPS 2025 · similarity 0.89)
• Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
• LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.88)
• Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.88)