Research map / Reinforcement learning / RL from feedback
UCPO: Uncertainty-Aware Policy Optimization
Xianzhou Zeng, Jing Huang, Chunmei Xie, Gongrui Nan, Siye Chen, Mengyu Lu, Weiqi Xiong, Qixuan Zhou, Junhao Zhang, Qiang Zhu, Yadong Li, Xingzhong Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add UCPO (TAD + DURA) over GRPO | Llama-3.1-8B-Instruct · Math & Text Reasoning (avg) | PAQ: 20.06 -> 28.45 | Improved |
| add UCPO over GRPO | Qwen3-8B · General Tasks (avg) | PAQ: 73.48 -> 79.68 | Improved |
| add UCPO over GRPO | Qwen3-8B · General Tasks (avg) | F1: 73.00 -> 70.79 | Hurt |
| replace fixed uncertainty reward (GRPO-UC r_u=0.5) with UCPO | Llama-3.1-8B-Instruct · Math & Text Reasoning (avg) | F1: 9.01 -> 22.65 | Improved |
| use fixed uncertainty reward r_u=0.8 instead of 0.2 | GRPO-UC on Llama · Math & Text Reasoning (avg) | F1: 13.31 -> 4.25 | Hurt |
| add Ternary Advantage Decoupling (TAD) (ablation) | UCPO on Llama-3.1-8B-Instruct · General Tasks | PAQ: 51.17 -> 58.58 | Improved |
| add DURA reward adjustment (ablation) | UCPO on Llama-3.1-8B-Instruct · General Tasks | F1: 23.41 -> 43.10 | Improved |
| add DURA reward adjustment (ablation) | UCPO on Llama-3.1-8B-Instruct · General Tasks | PAQ: 58.96 -> 58.58 | Hurt |
| add Non-Ternary Filtering (NTF) (ablation) | UCPO on Llama-3.1-8B-Instruct · Math & Text Reasoning | F1: 22.93 -> 22.65 | Improved |
| add Non-Ternary Filtering (NTF) (ablation) | UCPO on Llama-3.1-8B-Instruct · General Tasks | PAQ: 55.68 -> 58.58 | Improved |
| add Low-Resource Extensions (LRE) (ablation) | UCPO on Llama-3.1-8B-Instruct · General Tasks | F1: 39.99 -> 43.10 | Improved |
| raise DURA weight w from 1.0 to 2.0 (ablation) | UCPO on Llama-3.1-8B-Instruct · General Tasks (avg) | PAQ / F1: PAQ 58.58 -> 54.53; F1 43.10 -> 45.05 | Mixed |
Closest papers by meaning
- Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards (ICML 2026 · similarity 0.88)
- Ask a Strong LLM Judge when Your Reward Model is Uncertain (NeurIPS 2025 · similarity 0.88)
- Mitigating Reward Overoptimization via Lightweight Uncertainty Estimation (NeurIPS 2024 · similarity 0.87)
- Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.87)
- One-Way Policy Optimization for Self-Evolving LLMs (ICML 2026 · similarity 0.87)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
