Research map / Large language models / Alignment and preferences

Your Weak LLM is Secretly a Strong Teacher for Alignment

ICLR 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that a weak LLM (OPT-125M) trained on labeled HH-RLHF preferences can label unlabeled preference pairs so that DPO-aligned student models match or exceed human-label DPO, with supervisor size having little effect.

Problem

Alignment usually requires either expensive human preference labels (RLHF) or costly strong-LLM feedback (RLAIF); the middle ground of using a small, cheap weak LLM as the preference labeler for alignment has not been systematically evaluated, especially for generative tasks.

Approach

Train a weak LLM with DPO on a small labeled split, use its implicit DPO reward to relabel unlabeled response pairs, then DPO-align a larger student on the weakly labeled data; evaluate via generation quality (gold reward and GPT-4 win rate) rather than preference accuracy.

Key results

• Student models aligned with OPT-125M weak feedback match or exceed human-feedback DPO gold reward across OPT-1.3B to OPT-13B on HH-RLHF.
• Weak supervisors of 125M, 1.3B, and Llama-3-8B give nearly comparable alignment performance, and OPT-125M outperforms GPT-4 as a labeler in some cases.
• GPT-4 win-rate of weak-feedback policies against human-feedback policies approaches 50% across five student models.

Measured effects reported in the paper

Surprising findings

• Preference pairs where the weak LLM contradicts human labels still improve alignment (gold reward +1.79 over the untuned model), and in about 44% of such mismatched cases the weak-chosen response has higher gold reward.
• A 125M weak LLM can outperform GPT-4 as a preference labeler, suggesting task-specific training beats prompting a larger model.

Limitations

• Evaluation relies on a learned gold reward model and GPT-4 judgments as proxies for human quality, not real human evaluation.
• Experiments are limited to HH-RLHF and Reddit TL;DR with mostly OPT-family models and DPO.
• GPT-4 preference consistency drops on subtle-difference pairs, so the reliability of the judge itself is limited.

Experimental setup

Datasets: Anthropic HH-RLHF Reddit TL;DR

Benchmarks: Anthropic HH-RLHF test set Reddit TL;DR

Metrics: gold reward GPT-4 win-rate preference consistency

Baselines: DPO with human feedback GPT-4 as LLM labeler (RLAIF-style) OPT-1.3B supervisor Llama-3-8B supervisor

Builds on: DPO RLHF weak-to-strong generalization RLAIF

Closest papers by meaning

• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.92)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.92)
• A transfer learning framework for weak to strong generalization (ICLR 2025 · similarity 0.91)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.91)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.91)
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment (NeurIPS 2025 · similarity 0.90)