Research map / Large language models / Instruction and fine-tuning

IRIS: Interpolative Rényi Iterative Self-play for Large Language Model Fine-Tuning

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IRIS, a Rényi-divergence self-play fine-tuning objective built from two tilted risk terms with an adaptive order schedule, unifying SPIN/SPACE/SPIF as divergence regimes and reporting gains on Zephyr-7B and Qwen2.5-3B across ten benchmarks.

Problem

Existing self-play fine-tuning methods (SPIN, SPACE, SPIF, etc.) each fix a single divergence (KL, Jensen-Shannon, chi-squared), which gives no uniformly favorable learning dynamics across training stages as the model-target gap changes.

Approach

Replace the fixed divergence with a Rényi-divergence-based objective decomposed into two independent tilted risk terms (over annotated and synthetic data) whose exponential importance weights are controlled by order alpha, and adapt alpha across iterations based on the expected reward gap.

Key results

• IRIS reaches 44.57% average on Zephyr-7B at Iter 4, vs 43.56% for T-SPIN and 41.17% for SPIN.
• On Qwen2.5-3B-Instruct, IRIS reaches 56.18% average at Iter 4, vs 55.73% for T-SPIN.
• IRIS with 26k annotated samples reaches 42.53% after two iterations, above SFT on 200k (41.99%).

Surprising findings

• Fixed alpha=2.0 (chi-squared regime) gives the strongest initial gains but becomes unstable after Iter 2.
• Training five epochs on fixed synthetic data plateaus well below one iteration of regenerated data, suggesting regeneration rather than longer training drives gains.

Limitations

• Assumes a fixed target distribution throughout training.
• Evaluation restricted to single-turn generation on models up to 7B parameters.
• Adaptive schedule lacks formal regret bounds.

Experimental setup

Datasets: UltraChat200k

Benchmarks: GSM8K MATH MUSR MMLU MMLU-Pro GPQA HellaSwag WinoGrande BBH IFEval

Metrics: accuracy average score

Baselines: SFT SPIN SPACE T-SPIN SPIF

Builds on: SPIN SPACE SPIF T-SPIN Donsker-Varadhan variational representation

Closest papers by meaning

• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (ICML 2024 · similarity 0.93)
• Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs (NeurIPS 2025 · similarity 0.91)
• SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models (NeurIPS 2025 · similarity 0.90)
• Preserving Diversity in Supervised Fine-Tuning of Large Language Models (ICLR 2025 · similarity 0.87)
• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference (ICML 2024 · similarity 0.87)
• Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.87)