Research map / Large language models / Alignment and preferences
RLTHF: Targeted Human Feedback for LLM Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RLTHF, a human-AI hybrid that labels preference data with an LLM, then iteratively uses reward-distribution analysis to pick a small set of LLM-mislabeled samples for human correction, reaching near full-human alignment with 6-7% of human annotations.
Problem
Human preference annotation for RLHF is costly, while AI feedback (RLAIF) is unreliable on hard or customized samples; the paper asks how to reach full-human annotation quality with minimal human effort.
Approach
Train a reward model on LLM-labeled preferences, inspect the sorted reward-difference distribution to locate likely-mislabeled samples (via elbow/knee points), spend human annotation only there, and iterate with amplification and back-off hyperparameters while keeping confident LLM labels.
Key results
- On HH-RLHF, RLTHF raises reward model preference accuracy from GPT-4o's 74.7% to 89.6% using 6% human annotations, against 91.8% for the fully human-labeled set.
- On TL;DR, RLTHF raises accuracy from 78.8% to 88.0% with 7% human annotations, against 89.6% for full human labels.
- DPO on Qwen2.5-3B with RLTHF-curated data reaches a 58.1% win rate vs SFT on HH-RLHF (GPT-4o labeler), above the 55.7% of the fully human-annotated baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply targeted human feedback to GPT-4o labels using a 1/4 shard | RLTHF reward modeling · HH-RLHF test | preference accuracy: 74.7% to 89.6% | Helped |
| Apply targeted human feedback to GPT-4o labels using a 1/4 shard | RLTHF reward modeling · TL;DR test | preference accuracy: 78.8% to 88.0% | Helped |
| Use targeted human annotations instead of AI-only labels | Qwen2.5-3B DPO · HH-RLHF | win rate against SFT: 49.2 to 58.1 | Helped |
| Use targeted human annotations instead of random human annotations | Qwen2.5-3B DPO · HH-RLHF | win rate against SFT: 52.5 to 58.1 | Helped |
| Use targeted human annotations instead of full human annotations | Qwen2.5-3B DPO · HH-RLHF | win rate against SFT: 55.7 to 58.1 | Helped |
| Use targeted human annotations instead of AI-only labels | Qwen2.5-3B DPO · TL;DR | win rate against SFT: 59.2 to 62.3 | Helped |
| Use targeted human annotations instead of random human annotations | Qwen2.5-3B DPO · TL;DR | win rate against SFT: 59.8 to 62.3 | Helped |
| Use targeted human annotations instead of full human annotations | Qwen2.5-3B DPO · TL;DR | win rate against SFT: 60.2 to 62.3 | Helped |
| Remove human annotations (ablation) | RLTHF · HH-RLHF | preference accuracy: 87.7 to 75.7 | Hurt |
| Remove human annotations (ablation) | RLTHF · TL;DR | preference accuracy: 83.7 to 75.2 | Hurt |
| Remove amplification and back-off (ablation) | RLTHF · HH-RLHF | preference accuracy: 87.7 to 75.8 | Hurt |
| Remove amplification and back-off (ablation) | RLTHF · TL;DR | preference accuracy: 83.7 to 76.0 | Hurt |
Surprising findings
- DPO trained on RLTHF-curated labels outperforms DPO trained on fully human-annotated labels, which the authors attribute to back-off filtering of noisy human labels.
- Pure self-improvement with no human annotations fails to exceed the GPT baseline's preference accuracy.
Limitations
- Hyperparameters (amplification and back-off ratios) are set by heuristics and limited observations and need tuning per dataset.
- Human labels in public datasets are noisy, and the 'full human' reference uses label-flipped ground truth, so the oracle is itself constructed.
- Cost analysis relies on suggested crowdsourcing prices and assumptions that may understate actual cost.
Experimental setup
Datasets: HH-RLHF TL;DR
Benchmarks: HH-RLHF TL;DR AlpacaEval
Metrics: preference accuracy win rate return on investment (accuracy gain per unit human annotatio cost
Baselines: AI-only labeling (GPT-4o) AI-only labeling (GPT-4o mini) Random human annotation Full human annotation
Builds on: DPO RLAIF Bradley-Terry model PPO SER
Closest papers by meaning
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.93)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.92)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.92)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.92)
- SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.92)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.