Research map / Reinforcement learning / RL from feedback
Learning from negative feedback, or positive feedback or both
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMPO, an EM-derived preference optimization objective that decouples learning from accepted and rejected samples, enabling training from positive-only, negative-only, or both feedback types with a KL term that makes negative-only learning stable.
Problem
Preference optimization methods such as DPO require paired preferred/dis-preferred data, limiting use when only unpaired positive or negative feedback is available; prior EM-based RL-as-inference methods use only positive outcomes.
Approach
Extends EM-based policy optimization (MPO-style) by re-expressing the E-step variational distribution in terms of dis-preference probabilities, yielding a loss that maximizes likelihood of accepted samples and minimizes likelihood of rejected samples with a closed-form KL to the reference policy; a weighting parameter alpha trades off the two terms.
Key results
- On RGB Stacking offline RL, Accept+Reject+BC reaches reward 93 versus 24 for BC alone and 77 for Reject+BC.
- In language alignment with Gemma 2B, PMPO fine-tuned model wins side-by-side against base Gemma checkpoint as judged by GPT-4, with PMPO-AR learning faster than PMPO-A or PMPO-R and competitive with DPO.
- On DeepMind Control Suite, learning only from rejected samples (PMPO-R) requires beta above 1.0 for effective learning, while accept-only learning is insensitive to beta.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rejection loss to BC (ablation) | BC on the RGB Stacking dataset · RGB Stacking | average reward: 24 -> 77 | Helped |
| Add acceptance loss to BC (ablation) | BC on the RGB Stacking dataset · RGB Stacking | average reward: 24 -> 26 | No clear effect |
| Train BC only on accepted transitions (ablation) | BC on the RGB Stacking dataset · RGB Stacking | average reward: 24 -> 27 | No clear effect |
| Add both acceptance and rejection losses to BC (ablation) | BC on the RGB Stacking dataset · RGB Stacking | average reward: 24 -> 93 | Helped |
Surprising findings
- Learning only from negative feedback is stable when a sufficiently large KL term to the reference policy is included, and the KL term is needed specifically for negative-only learning.
- PMPO-AR reward-hacks the imperfect reward model mid-training in the language experiments, with a drop in independent benchmark performance.
Limitations
- Learning effectively from negative feedback requires a good estimate of the KL term, ideally in closed form; otherwise many reference samples are needed.
- Beta must be tuned, and the derivation suggests beta should approach zero as alpha approaches one, yet experiments keep alpha and beta independent.
- Language experiments rely on a learned reward model that can be hacked.
Experimental setup
Datasets: LMSYS-chat-1M DeepMind Control Suite RGB Stacking benchmark Rosenbrock Sphere Schwefel
Benchmarks: DeepMind Control Suite RGB Stacking Rosenbrock Sphere Schwefel
Metrics: average episode reward win rate vs base model
Baselines: MPO DPO IPO
Builds on: MPO DPO IPO CRR
Closest papers by meaning
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning (NeurIPS 2025 · similarity 0.88)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.88)
- Bayesian Optimization from Human Feedback: Near-Optimal Regret Bounds (ICML 2025 · similarity 0.88)
- Reward Learning from Multiple Feedback Types (ICLR 2025 · similarity 0.88)
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.88)
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.