Research map / Large language models / Alignment and preferences
Progressively Label Enhancement for Large Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PLL, a framework that couples principle-guided response generation with dynamically thresholded ranking and reward-weighted SFT losses for LLM alignment, evaluated on HH with LLaMA3 8B against SFT, PPO, DPO and RAFT.
Problem
RLHF is unstable and costly, and existing alternatives (SFT-based, RAFT, DPO) rely on large high-quality data and treat data generation and training as static, separate processes, so generated data is used inefficiently.
Approach
Generate both plain and principle-guided responses per query, use a decaying threshold on their reward-score gap to choose between a ranking loss (when the principle-guided response is clearly better) and reward-normalized weighted SFT (when they are similar), so all generated data contributes to training.
Key results
- On HH with LLaMA3 8B, OURS obtains RM -0.75 versus -0.82 for RAFT, -0.87 for PPO, -0.96 for DPO and -1.34 for SFT.
- OURS PPL is 8.45, comparable to SFT (8.42) and RAFT (8.52), while PPO and DPO have PPL around 16.
- Claude Sonnet and five human annotators' win rates on 50 test queries are reported as favoring OURS over baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use progressively label-enhanced training instead of BASE | Llama3 8B · HH test set | RM: -1.80 -> -0.75 | Helped |
| Use progressively label-enhanced training instead of SFT | Llama3 8B · HH test set | RM: -1.34 -> -0.75 | Helped |
| Use progressively label-enhanced training instead of PPO | Llama3 8B · HH test set | RM: -0.87 -> -0.75 | Helped |
| Use progressively label-enhanced training instead of DPO | Llama3 8B · HH test set | RM: -0.96 -> -0.75 | Helped |
| Use progressively label-enhanced training instead of RAFT | Llama3 8B · HH test set | RM: -0.82 -> -0.75 | Helped |
| Use progressively label-enhanced training instead of SFT | Llama3 8B · HH test set | PPL: 8.42 -> 8.45 | No clear effect |
Surprising findings
- The principle-guided and original reward gap shrinks over training, suggesting the model internalizes the principles without explicit prompting.
Limitations
- Results are on a single dataset (HH) and a single base model (LLaMA3 8B).
- Theoretical analysis relies on strong assumptions (bounded density, reward range in [0,1], error bound assumption on the model).
- Evaluation uses a small 50-query subset for win-rate judgments and only five annotators.
Experimental setup
Datasets: Anthropic HH
Benchmarks: Anthropic HH
Metrics: perplexity reward margin win rate
Baselines: SFT PPO DPO RAFT
Builds on: RLHF RAFT RRHF LIMA self-align
Closest papers by meaning
- Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.90)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.90)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.89)
- Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models (NeurIPS 2025 · similarity 0.89)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.89)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.