Research map / Reinforcement learning / RL from feedback
Can You Rely on Synthetic Labellers in Preference-Based Reinforcement Learning? It’s Complicated
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares policies trained on synthetic (oracle) versus crowd-sourced human preference labels for walker stand/walk/run in PbRL, finding synthetic labels are a reasonable proxy early but diverge at the end of training.
Problem
Whether synthetic oracle preference labellers, used almost exclusively in preference-based RL research, faithfully reflect real human preference labels and the resulting policy performance.
Approach
Runs a controlled comparison in which the same queried behavior pairs are labelled either by a crowd of ~50 humans per pair or by an oracle using the DMC ground-truth reward, and measures policy normalized returns, label matching, and inter-annotator agreement across feedback rounds.
Key results
- Policies trained on crowd labels perform nearly identically to synthetic-label policies until the final feedback round (step 190k), where synthetic policies outperform (e.g., walk 0.77 vs 0.71 NR over 500k steps for synthetic vs human, stand 0.82 vs 0.60).
- Early in training (first feedback round, 20k steps), human feedback yields higher normalized returns than synthetic (stand 0.66 vs 0.54, walk 0.87 vs 0.69, run 1.01 vs 0.96).
- Training on the highest inter-annotator agreement pairs does not beat random subsampling (e.g., stand 0.55 vs 0.66 random), while lowest agreement is clearly worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use human rather than synthetic preference labels during the first feedback round | SAC with learned reward · stand, walk, and run | normalized returns: stand: 0.54 -> 0.66; walk: 0.69 -> 0.87; run: 0.96 -> 1.01 | Helped |
| use synthetic rather than human preference labels in the final training period | SAC with learned reward · stand, walk, and run, 190–500k steps | normalized returns: stand: 0.61 -> 0.96; walk: 0.80 -> 0.91; run: 0.53 -> 0.60 | Helped |
| use synthetic rather than human preference labels throughout training | SAC with learned reward · stand, walk, and run, 0–500k steps | normalized returns: stand: 0.60 -> 0.82; walk: 0.71 -> 0.77; run: 0.48 -> 0.53 | Mixed |
| select samples with highest inter-annotator agreement instead of random samples (ablation) | SAC with human feedback · stand and walk | normalized returns: stand: 0.66 -> 0.55; walk: 0.87 -> 0.56 | Hurt |
| select samples with highest inter-annotator agreement instead of random samples (ablation) | SAC with human feedback · run | normalized returns: 1.01 -> 1.08 | No clear effect |
| select samples with lowest inter-annotator agreement instead of random samples (ablation) | SAC with human feedback · stand, walk, and run | normalized returns: stand: 0.66 -> 0.44; walk: 0.87 -> 0.49; run: 1.01 -> 0.87 | Hurt |
Surprising findings
- Synthetic labels never produce ties, while crowd labels frequently indicate ties or weak preferences, yet human feedback yields better early policies.
- Selecting training samples with highest inter-annotator agreement does not improve policy performance over random subsampling.
Limitations
- Only three DMC walker tasks are studied, limiting generality.
- Crowd labels come from an anonymous crowdsourcing platform with no participant demographic information.
Experimental setup
Datasets: DMC walker stand/walk/run behavior pairs labelled by crowd and oracle
Benchmarks: DMC walker stand DMC walker walk DMC walker run
Metrics: normalized returns Fleiss' kappa normalized L2 agreement label matching rate
Baselines: SAC on DMC ground-truth reward SAC on synthetic-feedback reward
Builds on: PEBBLE REED SAC SimSiam
Closest papers by meaning
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.87)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.87)
- HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages (NeurIPS 2025 · similarity 0.87)
- SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.86)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.86)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.