Research map / Large language models / Instruction and fine-tuning

A transfer learning framework for weak to strong generalization

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Frames weak-to-strong generalization as transfer learning over a latent-concept mixture model, shows naive fine-tuning on weak labels provably has poor risk, and proposes ICL-based output refinement that resamples labels from the strong model to recover the target concept.

Problem

Aligning stronger LLMs using labels from weaker models without degrading the strong model's capabilities (weak-to-strong generalization); naive fine-tuning on weak labels may be limited.

Approach

Casts weak-to-strong generalization as transferring a latent concept prior from a weak to a strong model; assumes the target conditional lies in the convex hull of the source mixture, so the strong model's in-context learning can refine weak labels toward the target concept before fine-tuning.

Key results

• Naive fine-tuning on weak pirate-persona labels lowers test-time content score below the base GPT-3.5-Turbo, and the degradation worsens as weak label quality drops; ICL refinement keeps content scores near or above base.
• On GSM8K/MATH with GPT-3.5-Turbo and all three weak labelers, naive fine-tuning fails to beat the unchanged model, while ICL-refined labels improve reasoning.
• For GPT-4o-mini, refinement outperforms naive fine-tuning, but gains over the base model are limited.

Measured effects reported in the paper

Surprising findings

• Naive fine-tuning on weak labels often reduces the strong model's accuracy below its unchanged baseline, so weak-to-strong generalization has not truly occurred by that measure.
• The auxiliary confidence loss improves accuracy but sharply reduces style transfer.

Limitations

• Theory relies on a strong linear mixture-of-regressions assumption and iid treatment of ICL examples.
• For GPT-4o-mini, gains from refined labels remain limited relative to the base model.
• Evaluation uses small tiny benchmarks (100 questions) and GPT-4o as judge.

Experimental setup

Datasets: Dolly GSM8K MATH AlpacaEval 2.0 (tiny) TruthfulQA (tiny)

Benchmarks: tinyAlpacaEval tinyTruthfulQA GSM8K MATH science explanation questions

Metrics: content score style score answer accuracy (GPT-4o judged 0/1)

Baselines: naive fine-tuning on weak labels unchanged strong model bootstrapping (weak to GPT-3.5-Turbo to GPT-4o-mini) auxiliary confidence loss

Builds on: weak-to-strong generalization in-context learning as implicit Bayesian inference mixture-of-regressions transformer

Closest papers by meaning

• Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.91)
• Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.91)
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (ICML 2024 · similarity 0.90)
• Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.90)
• Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization (AAAI 2025 · similarity 0.89)
• The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge (NeurIPS 2026 · similarity 0.88)