Research map / Large language models / Language, safety and interpretability

Quantifying the Gain in Weak-to-Strong Generalization

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives a representation-theoretic theory showing that, for least-squares regression over convex function classes, a strong model trained on weak labels improves over the weak model by at least its misfit with the weak model, and validates this on synthetic MLP and molecular/NLP regression experiments.

Problem

Weak-to-strong generalization (Burns et al.) lacks theoretical justification: when does a strong model fine-tuned on weak-model labels provably beat its weak supervisor, and by how much?

Approach

Model weak and strong models as different representation maps with a shared linear/convex finetuning class; the strong model's weakly supervised solution is the projection of the weak model onto a convex set, so by the Pythagorean property the true error of the strong model is at most the weak error minus the misfit. Adds finite-sample and non-realizable extensions.

Key results

• Theorem: in the realizable convex setting, error of weakly supervised strong model on true task is at most weak model error minus misfit, proven via Pythagorean projection argument.
• Synthetic and QSAR experiments show gain in accuracy aligns closely with misfit across tasks and splits (plots).
• Heuristic choosing the weak model minimizing (weak error - misfit) picked the lowest-true-error strong model on Lipop (table), working well on ESOL but not exactly on FreeSolv.

Measured effects reported in the paper

Surprising findings

• In low-sample representation learning regimes, the less expressive model yields better representations and effectively becomes the strong model (roles reversed).
• Empirical gain-misfit relationship appears to hold even when the finetuning class is non-convex (tanh, relu, sigmoid activations), beyond what theory covers.

Limitations

• Theorems require the strong model's finetuning class to be convex and squared loss; do not cover classification.
• Error terms in non-realizable finite-sample bound can be non-negligible, as seen when upper bounds become negative in experiments.
• Real-world experiments are small-scale regression tasks rather than large modern LLMs.

Experimental setup

Datasets: synthetic Gaussian data ESOL FreeSolv Lipop GuacaMol Feedback Prize - English Language Learning French public service reviews

Benchmarks: ESOL FreeSolv Lipop synthetic regression Feedback Prize ELL essay scoring French public service reviews

Metrics: squared error gain in accuracy (weak error minus strong-on-true error) misfit

Baselines: GPT-2-style weak supervisor (Burns et al. setup)

Builds on: Burns et al. weak-to-strong generalization MolBERT boosting

Closest papers by meaning

• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (ICML 2024 · similarity 0.93)
• Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.93)
• Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions (ICML 2025 · similarity 0.92)
• A transfer learning framework for weak to strong generalization (ICLR 2025 · similarity 0.91)
• On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.90)
• Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization (ICLR 2025 · similarity 0.90)