Research map / Reinforcement learning / RL from feedback
OnePO: Direct One-stage Policy Optimization for SFT-free Domain Adaptation
Junying Chen, Xinyuan Xie, Ziniu Li, Benyou Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add probability floor (ablation) | OnePO · HealthBench Total | score (%): 65.4 -> 49.1 when removed | Improved |
| add gradient rescaling (ablation) | OnePO · HealthBench Total | score (%): 65.4 -> 57.3 when removed | Improved |
| add Teacher Retirement (ablation) | OnePO · HealthBench Total | score (%): 65.4 -> 52.7 when removed | Improved |
| add Teacher Retirement (ablation) | OnePO · HealthBench Hard | score (%): 39.1 -> 18.6 when removed | Improved |
| add probability floor (ablation) | OnePO · MMLU-Pro (Med) | score (%): 81.2 -> 73.1 when removed | Improved |
| use strict max retirement threshold instead of looser or none (ablation) | OnePO · training curves | evaluation score: Max > Mean > Min > None | Improved |
| use OnePO instead of SFT+RL (DeepSeek-V3.2-Thinking teacher) | Qwen3-8B-Base · HealthBench Total | score (%): 64.5 -> 67.2 | Improved |
| use OnePO instead of SFT+RL (GPT-5-Chat teacher) | Qwen3-8B-Base · HealthBench Total | score (%): 63.6 -> 65.4 | Improved |
| add teacher guidance via OnePO over pure RL | Qwen3-8B-Base · MedXpertQA (Text) | score (%): 20.0 -> 25.9 | Improved |
| add teacher guidance via SFT+RL over pure RL | Qwen3-8B-Base · Medbullets (5-option) | score (%): 48.1 -> 61.9 | Improved |
| use weaker teacher (Qwen3-8B-Thinking) instead of DeepSeek-V3.2-Thinking | OnePO · HealthBench Total | score (%): 67.2 -> 66.0 | No clear effect |
| use mixed verifiable and rubric reward instead of rubric-only (ablation) | OnePO · HealthBench Total | score (%): 63.7 -> 65.4 | Improved |
Closest papers by meaning
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.87)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Pre-Trained Multi-Goal Transformers with Prompt Optimization for Efficient Online Adaptation (NeurIPS 2024 · similarity 0.87)
- LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL (NeurIPS 2026 · similarity 0.86)
- Cross-Domain Policy Adaptation by Capturing Representation Mismatch (ICML 2024 · similarity 0.86)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
