Research map / Reinforcement learning / RL from feedback
Positive–Unlabeled Reinforcement Learning Distillation for On-Premise Small Models
Zhiqiang Kou, Junyang Chen, Xin-Qiang Cai, Xiaobo Xia, Ming-Kun Xie, Dong-Dong Wu, Biao Liu, Yuheng Jia, Xin Geng, Masashi Sugiyama, Tat-Seng Chua
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LDL-GRPO instead of SFT only | LLaMA3-8B-Instruct · PBFG | raw win rate vs GPT-4o: 0.482 -> 0.834 | Improved |
| replace scalar-reward Anchor-GRPO with label-distribution LDL-GRPO (ablation) | LLaMA3-8B-Instruct · PBFG | raw win rate: 0.673 -> 0.834 | Improved |
| replace Anchor-GRPO with LDL-GRPO (ablation) | LLaVA-7B · A-OKVQA-MC | raw win rate: 0.090 -> 0.146 | Improved |
| use Anchor-GRPO instead of SFT (ablation) | LLaVA-7B · A-OKVQA-MC | raw win rate: 0.131 -> 0.090 | Hurt |
| continue SFT on teacher outputs (SFT->SFT) instead of preference stage | Qwen2.5-VL-7B / LLaMA3-8B · writing tasks | raw win rate: 0.538 -> 0.352 on CFCW (Qwen2.5-VL column) | Hurt |
| induce preferences from one anchor and multiple candidates instead of a single teacher/student pair | DPO · six tasks | win rate | Improved |
| use LDL-GRPO instead of AnchorRank-DPO | LLaMA3-8B-Instruct · PBFG | raw win rate: 0.814 -> 0.834 | Improved |
| vary sampling temperature from 0.6 to 1.5 (ablation) | LDL-GRPO · CountProb (LLaMA3-8B) | win rate | No clear effect |
| increase KL penalty beta to large values (ablation) | LDL-GRPO · CountProb (LLaMA3-8B) | win rate | Hurt |
| add the two-stage SFT-to-LDL-GRPO switch | LDL-GRPO · CountProb, A-OKVQA-RG | training loss | Improved |
Closest papers by meaning
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.88)
- RLCD: Reinforcement Learning from Contrastive Distillation for LM Alignment (ICLR 2024 · similarity 0.88)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.87)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.87)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.87)
- Preference Distillation via Value based Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
