Research map / Large language models / Alignment and preferences
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that strong GPT-4-family models finetuned on labels from much weaker models often outperform their supervisors, and that simple methods (auxiliary confidence loss, bootstrapping, generative finetuning) recover more of the gap.
Problem
Superalignment: can weak supervisors (humans or small models) elicit the full capabilities of much stronger models, when naive finetuning on weak labels imitates the weak supervisor's errors?
Approach
Study weak-to-strong learning as an empirical analogy by finetuning large pretrained GPT-4-family base models on labels produced by small weak models, measure performance gap recovered (PGR), and propose an auxiliary confidence loss that lets the strong student confidently disagree with weak labels.
Key results
- On NLP tasks, GPT-4 finetuned with GPT-2-level weak labels and the auxiliary confidence loss recovers close to GPT-3.5-level performance; median PGR rises from about 25% to nearly 80% with the smallest supervisor and largest student.
- Naive weak-to-strong finetuning recovers roughly 10% of the gap on the ChatGPT reward modeling task, and PGR on chess decreases with student size for a fixed supervisor.
- Finetuning on weak labels then linear probing on ground truth reaches 78% average accuracy, closing 60% of the gap to ground-truth finetuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add auxiliary confidence loss to weak-label finetuning | GPT-4 student trained with GPT-2-level supervision · NLP tasks | performance gap recovered: about 25% to nearly 80% | Helped |
| Add auxiliary confidence loss to weak-label finetuning | Strong students trained with weak supervision · NLP tasks | accuracy | Mixed |
| Bootstrap supervision through intermediate model sizes | Strong students trained with weak supervision · Chess puzzles | performance gap recovered | Helped |
| Bootstrap supervision through intermediate model sizes | Strong students trained with weak supervision · NLP tasks | performance gap recovered | No clear effect |
| Bootstrap supervision through intermediate model sizes | Strong students trained with weak supervision · ChatGPT reward modeling | performance gap recovered | No clear effect |
| Add unsupervised generative finetuning on reward-modeling data | Strong students trained with weak supervision · ChatGPT reward modeling | performance gap recovered: approximately 10-20% | Helped |
| Combine generative finetuning with ground-truth early stopping | Strong students trained with weak supervision · ChatGPT reward modeling | performance gap recovered: approximately 30-40% | Helped |
| Finetune on weak labels, then train a linear probe on ground-truth labels | GPT-4 family model · NLP tasks, subset average | accuracy: 78% | Helped |
| Use LP-FT: train a linear probe, then finetune all layers | Strong model trained with weak supervision · Weak-to-strong learning | performance | No clear effect |
| Apply weight regularization with stronger weight decay or dropout | Strong model trained with weak supervision · Weak-to-strong learning | performance | No clear effect |
| Apply LoRA and sweep the rank | Strong model trained with weak supervision · Weak-to-strong learning | performance | No clear effect |
| Add consistency loss between original and rephrased inputs | Strong model trained with weak supervision · Selected NLP datasets | performance | No clear effect |
Surprising findings
- Larger strong students agree less with weak supervisor errors (inverse scaling in imitation), despite being trained to imitate the supervisor.
- Strong students trained on weak labels sometimes underperform the zero-shot baseline on chess puzzles.
Limitations
- Remaining disanalogies: imitation saliency and pretraining leakage may make results optimistic relative to superhuman alignment.
- Methods do not work consistently; reward modeling remains far from full gap recovery.
- Reward modeling uses a proprietary dataset and GPT-4 family models are not publicly available.
Experimental setup
Datasets: BoolQ CosmosQA DREAM ETHICS FLAN ANLI R2 GLUE CoLA GLUE SST-2 HellaSwag MCTACO OpenBookQA PAWS QuAIL
Benchmarks: 22 NLP classification datasets chess puzzles (lichess) ChatGPT reward modeling test set
Metrics: test accuracy performance gap recovered (PGR) student-supervisor agreement
Baselines: naive finetuning on weak labels 5-shot prompting zero-shot prompting linear probing ground truth finetuning
Builds on: RLHF self-training conditional entropy minimization knowledge distillation
Closest papers by meaning
- Debate Helps Weak-to-Strong Generalization (AAAI 2025 · similarity 0.95)
- Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.93)
- Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization (ICLR 2025 · similarity 0.93)
- Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions (ICML 2025 · similarity 0.93)
- Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization (AAAI 2025 · similarity 0.92)
- The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.