Research map / Reinforcement learning / RL from feedback
Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Uni-RLHF, a multi-feedback annotation platform, crowdsourced feedback datasets (~15M steps over 30 tasks), and offline RLHF baselines (IQL/CQL/TD3BC with MLP/CNN/Transformer reward models) for benchmarking learning from human feedback.
Problem
Studying RLHF with diverse human feedback is hampered by the lack of standardized annotation platforms, unified benchmarks, and reusable real crowdsourced feedback datasets; prior benchmarks rely on scripted teachers.
Approach
A universal annotation platform supporting five feedback types (comparative, attribute, evaluative, visual, keypoint) with a standardized encoding format, paired with a crowdsourced annotation pipeline with ex-ante filters and decoupled offline RLHF baselines evaluated against oracle and script-teacher labels.
Key results
- On SMARTS, IQL with crowdsourced labels reaches average success rate 0.71 versus 0.70 for the Oracle reward and 0.63 for script-teacher labels.
- Crowdsourced annotation pipeline with examples and filters reaches 98% agreement with expert labels on left-c.
- On 5 Atari tasks, DiscreteCQL-CS outperforms DiscreteCQL-ST on all tasks, e.g., Breakout 33.5 vs 3.0 and Qbert 11301.7 vs 8170.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use crowdsourced rather than synthetic comparative feedback | DiscreteCQL · Atari tasks | score: Boxing-m: 62.4±7.8 -> 64.5±10.6; Breakout-m: 3.0±1.0 -> 33.5±10.8; Enduro-m: 189.5±8.3 -> 191.5±8.3; Pong-m: 18.5±2.2 -> | Helped |
| use crowdsourced rather than hand-designed task rewards | IQL · SMARTS | success rate: cruise: 0.71±0.03 -> 0.62±0.03; left-c: 0.53±0.03 -> 0.70±0.03; cutin: 0.85±0.04 -> 0.80±0.03; average: 0.70 -> 0.71 | Mixed |
| add predicted keypoints as subgoals (ablation) | TD3BC · maze2d-medium-v1 and maze2d-large-v1 | normalized score: maze2d-medium-v1: 59.5±36.3 -> 131.7±33.3; maze2d-large-v1: 97.1±25.4 -> 150.6±60.2 | Helped |
| replace random query sampling with disagreement sampling (ablation) | IQL-MLP-CrowdSourced · antmaze-medium-diverse-v2 | normalized score: 46.95±11.91 -> 64.43±6.57 | Helped |
| replace random query sampling with disagreement sampling (ablation) | IQL-MLP-CrowdSourced · hopper-medium-replay-v2 | normalized score: 42.75±5.27 -> 63.28±8.58 | Helped |
| replace random query sampling with entropy sampling (ablation) | IQL-MLP-CrowdSourced · antmaze-medium-diverse-v2 | normalized score: 46.95±11.91 -> 62.69±7.51 | Helped |
| replace random query sampling with entropy sampling (ablation) | IQL-MLP-CrowdSourced · hopper-medium-replay-v2 | normalized score: 42.75±5.27 -> 44.53±6.83 | No clear effect |
Surprising findings
- Crowdsourced labels outperform script-teacher labels in most settings, which the authors attribute to synthetic labels being shortsighted.
- CQL occasionally collapses under crowdsourced labels (e.g., CQL-CS-MLP on hopper-medium-replay).
Limitations
- Large-scale experiments only use comparative feedback; other feedback types are evaluated only as primers.
- Lacks deep analysis of human irrationality and bias.
- Results averaged over only 3 seeds.
Experimental setup
Datasets: D4RL D4RL_Atari SMARTS DMControl walker attribute feedback Maze2d
Benchmarks: D4RL D4RL_Atari SMARTS
Metrics: expert-normalized score success rate speed comfort (jerk) annotation agreement with experts
Baselines: Oracle (hand-designed rewards) ST (script teacher labels) IQL CQL TD3BC Preference Transformer
Builds on: Preference Transformer Implicit Q-Learning Conservative Q-Learning TD3BC D4RL
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.92)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.92)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.92)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.92)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- How to Evaluate Reward Models for RLHF (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.