Research map / Large language models / Alignment and preferences
Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
Yidong Wang, Xin Wang, Cunxiang Wang, Junfeng Fang, Qiufeng Wang, Jianing Chu, Xuran Meng, Shu-Xun Yang, Andrew Zhuoer Feng, Libo Qin, Wei Ye, Shikun Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-rewarding pairs with Temporal SR (anchored rejection + future-guided chosen), half the iterations | Llama3.1-8B · AlpacaEval 2.0 | win rate (%): 19.69 -> 29.44 | Improved |
| Add Anchored Rejection (past initial model rejected responses) only (ablation) | Self-Rewarding Llama3.1-8B · AlpacaEval 2.0 | win rate (%): 19.69 -> 29.06 | Improved |
| Add Future-Guided Chosen only (ablation) | Self-Rewarding Llama3.1-8B · Arena-Hard-v0.1 | score: 6.7 -> 14.0 | Improved |
| Continue Future-only training for another iteration (ablation) | Temporal SR w/o Past · AlpacaEval 2.0 | win rate (%): 22.05 -> 21.49 | No clear effect |
| Add Future-Guided Chosen on top of Anchored Rejection (ablation) | Temporal SR Llama3.1-8B · Arena-Hard-v0.1 | score: 13.4 (w/o Future, iter 3) vs 14.6 (full, iter 1) | Improved |
| Use stronger external judge AutoJ-13B instead of self-judge (ablation) | Self-Rewarding and Temporal SR · AlpacaEval 2.0 | win rate (%): SR 19.7 -> 28.9; Temporal SR 29.4 -> 28.6 | Improved |
| Use Temporal SR across judge choices (ablation) | Llama3.1-8B · AlpacaEval 2.0 / Arena-Hard | win rate and score: Arena-Hard with AutoJ-13B: 22.4 (Temporal SR) vs 15.2 | Improved |
| Apply Temporal SR to Qwen2.5-7B | Qwen2.5-7B · Arena-Hard-v0.1 | score: 21.5 -> 34.4 | Improved |
| Apply Temporal SR to Mistral-7B | Mistral-7B · MT-Bench | average score: 5.68 -> 5.49 | Mixed |
| Apply Temporal SR to Llama3.2-3B | Llama3.2-3B · MT-Bench | average score: 4.03 -> 4.32 | Improved |
| Apply Temporal SR to Llama3.1-70B | Llama3.1-70B · AlpacaEval 2.0 | win rate (%): 32.91 -> 33.66 | Improved |
| Add temporal offline references to GRPO | Qwen2.5-7B-Math · AIME25 | accuracy: 8.5 -> 10.8 | Improved |
Closest papers by meaning
- Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.91)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.91)
- Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.90)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.90)
- Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.90)
- Self-Rewarding Language Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
