Research map / Large language models / Agents, code and math
SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPaR, a self-play framework where an LLM acts as actor and refiner, using tree-search self-refinement of failed responses to build minimal-variation preference pairs for DPO, improving instruction following over three iterations.
Problem
Preference pairs built from independently sampled responses contain irrelevant content variations that obscure the key differences determining instruction-following success, hampering preference learning.
Approach
Instead of sampling independent responses, the model refines its own failed responses via tree search (BFS/DFS) guided by its own judge, so refined and original responses differ only in instruction-relevant aspects; actor is trained with DPO and refiner with RFT in iterative self-play.
Key results
- SPaR-8B-DPO-iter3 reaches 81.8 average IFEval score, surpassing GPT-4-Turbo (81.3) and LLaMA3-8B-Instruct (77.9).
- Tree-search inference on SPaR-8B-DPO-iter3 raises IFEval average to 83.7.
- SPaR-70B-DPO-iter3 improves LLaMA3-70B-Instruct IFEval average from 83.4 to 86.1 and FollowBench average from 70.8 to 74.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with SPaR refinement preference pairs for three iterations | LLaMA3-8B-Instruct · IFEval Avg. | accuracy: 77.9 -> 81.8 | Helped |
| apply tree-search refinement at inference | SPaR-8B-DPO-iter3 · IFEval Avg. | accuracy: 81.8 -> 83.7 | Helped |
| add tree search (ablation) | SPaR actor · IFEval Prompt (S) | accuracy: -2.0 | Helped |
| add iterative training (ablation) | SPaR actor · FollowBench SSR Avg. | SSR: -2.0 | Helped |
| add refinement preference data (ablation) | SPaR actor · FollowBench SSR Avg. | SSR: -3.1 | Helped |
| add tree search (ablation) | SPaR refiner · LLMBar Adversarial | accuracy: -4.3 | Helped |
| add iterative training (ablation) | SPaR refiner · LLMBar Adversarial | accuracy: -1.7 | Helped |
| replace greedy refinement with BFS | SPaR-8B-RFT-iter3 · Refinement test set | Acc-GPT: 69.5 -> 79.0 | Helped |
| replace independently sampled preference pairs with refinement pairs | instruction-following preference training · Character Sequence Generation and Start/End Story Generation | accuracy | Helped |
Surprising findings
- Interfering-factor pairs (e.g., uppercase ratio or diverging stories) can underperform the original model, e.g., story generation with independent pairs drops below baseline.
- The refiner's refinement accuracy is much higher when self-judged (90.5%) than when judged by GPT-4o (79.0%), indicating self-evaluation bias.
Limitations
- Self-evaluation bias: refiner judges its own refinements more favorably than GPT-4o does.
- Relies on a strong LLM (GPT-4o-Mini) for bootstrapping SFT data and on the model's own judge for preference labels.
Experimental setup
Datasets: Infinity-Instruct IFEval FollowBench LLMBar GSM8K TriviaQA MMLU HumanEval
Benchmarks: IFEval FollowBench LLMBar GSM8K TriviaQA MMLU HumanEval
Metrics: prompt-level accuracy (loose/strict) instruction-level accuracy SSR (FollowBench) accuracy F1 refinement success rate
Baselines: AutoIF SELF Self-Rewarding Meta-Rewarding Humpback GPT-4-Turbo
Builds on: DPO Self-Rewarding Meta-Rewarding SELF Self-consistency
Closest papers by meaning
- Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models (ICLR 2025 · similarity 0.90)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.90)
- DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation (AAAI 2024 · similarity 0.89)
- LESS: Selecting Influential Data for Targeted Instruction Tuning (ICML 2024 · similarity 0.89)
- Evaluating the Zero-shot Robustness of Instruction-tuned Language Models (ICLR 2024 · similarity 0.89)
- Improving Instruction-Following in Language Models through Activation Steering (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.