Research map / Large language models / Alignment and preferences
$\textit{S}$-SPPO: Semantic-Calibrated Self-Play Preference Optimization
Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, ZhengXiao He, XUANZHAO DONG, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add supervision calibration (semantic gating of win-rate targets) (ablation) | SPPO, Llama-3-8B · AlpacaEval 2.0 LC win rate, Iter 3 | LC win rate: 44.79 -> 46.16 | Improved |
| add representation calibration (latent repulsion) (ablation) | SPPO, Llama-3-8B · AlpacaEval 2.0 LC win rate, Iter 3 | LC win rate: 44.79 -> 45.20 | Improved |
| combine both calibrations (ablation) | SPPO, Llama-3-8B · AlpacaEval 2.0 LC win rate, Iter 3 | LC win rate: 44.79 -> 47.46 | Improved |
| use S-SPPO instead of SPPO | SPPO, Llama-3-8B · Arena-Hard-Auto | win rate: 29.8 -> 31.5 | Improved |
| use S-SPPO instead of SPPO | SPPO, Llama-3-8B · MT-Bench | average score: 7.93 -> 8.22 | Improved |
| use S-SPPO instead of SPPO | SPPO, Mistral-7B · AlpacaEval 2.0 (GPT-4 annotator) | LC win rate: 28.53 -> 30.84 | Improved |
| use S-SPPO instead of SPPO in Mistral-7B iteration 1 | SPPO, Mistral-7B · AlpacaEval 2.0 (GPT-4 annotator) | LC win rate: 24.79 -> 24.28 | Hurt |
| swap semantic encoder (jina, nomic, MiniLM) (ablation) | S-SPPO · AlpacaEval 2.0 LC win rate, Iter 3 | LC win rate: 46.9 to 47.7 | No clear effect |
| vary lambda of representation calibration (ablation) | S-SPPO · AlpacaEval 2.0 | win rate | Mixed |
| use S-SPPO instead of SPPO | SPPO, Llama-3-8B · GSM8K strict-match | accuracy: 78.2% -> 78.9% | Improved |
| add S-SPPO calibration overhead | SPPO, Llama-3-8B · compute per iteration | peak GPU memory: 67.0GB -> 72.8GB | Hurt |
Closest papers by meaning
- Self-Play Preference Optimization for Language Model Alignment (ICLR 2025 · similarity 0.93)
- AMPO: Active Multi Preference Optimization for Self-play Preference Selection (ICML 2025 · similarity 0.90)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.90)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.90)
- Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment (ICLR 2025 · similarity 0.89)
- 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
