Research map / Label-efficient and robust learning / Recognition and long-tail
Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes the misfit-gain characterization of weak-to-strong generalization from squared loss to Bregman-divergence losses, and to classification via KL/cross-entropy using convex combinations of k logistic heads.
Problem
Prior misfit-based guarantees for weak-to-strong generalization hold only for squared-loss regression with convex strong classes; it is unclear whether they extend to classification (cross-entropy) and non-convex strong model classes.
Approach
The generalized Pythagorean inequality holds for Bregman divergences, so any Bregman-loss task gets a misfit-gain inequality; cross-entropy equals KL plus an entropy constant, enabling a recipe that finetunes a convex combination of k logistic heads by minimizing reverse KL to weak labels, with an O(sqrt(c/k)) error term for non-convex classes.
Key results
- On NLP datasets with gpt2 weak and gpt2-medium/large/xl strong, test loss of the strong model is always below the weak model's, and misfit tracks loss gain across datasets.
- For ViT-B/8 strong on ImageNet, the authors' weak-to-strong accuracy is reported as better than the 64.2% reported in OpenAI's Table 3 for the same setting.
- With the realizability fix (evaluating against best strong model), the misfit more faithfully matches the gain, while raw evaluation shows misfit consistently exceeding the gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reverse KL minimization with a convex combination of logistic heads instead of standard cross-entropy with a single head | ViT-B/8 weak-to-strong training · ImageNet | accuracy | Helped |
| Increase strong model size from gpt2-medium to gpt2-large to gpt2-xl | Weak-to-strong classification · BoolQ, SciQ, CosmosQA, Amazon Polarity | test loss | Helped |
| Increase strong model size from gpt2-medium to gpt2-large to gpt2-xl | Weak-to-strong classification · CosmosQA, Amazon Polarity, SciQ | test accuracy | Mixed |
| Increase strong-model misfit onto weak labels | Weak-to-strong classification · CIFAR-10, ImageNet | test loss | Helped |
| Increase strong-model misfit onto weak labels | Weak-to-strong classification · CIFAR-10, ImageNet | test accuracy | Helped |
| Increase the number of logistic heads in the convex combination (ablation) | Weak-to-strong classification with gpt2-xl or ViT-B/8 · All evaluated datasets except ImageNet | difference between misfit and gain | Helped |
| Increase the number of logistic heads in the convex combination (ablation) | Weak-to-strong classification with ViT-B/8 · ImageNet | difference between misfit and gain | No clear effect |
| Evaluate loss relative to the best strong model trained on true labels rather than the ground-truth target (ablation) | Weak-to-strong evaluation · NLP and Vision datasets | alignment between misfit and gain | Helped |
| Increase the number of logistic heads in the convex combination (ablation) | Synthetic weak-to-strong classification · Synthetic data, c=2, 10, 50, 100 | agreement between misfit and gain | Mixed |
Surprising findings
- The recipe minimizes KL with the strong model's output in the first argument, the reverse of standard cross-entropy supervision, yet still yields provable gain.
- Measured gain is consistently smaller than misfit when evaluated against ground truth, attributed to violations of realizability.
Limitations
- Theory assumes realizability (target exactly representable by strong model), which does not hold empirically.
- The reverse KL objective is non-convex, making optimization harder and lowering accuracies versus prior work.
- Bound for non-convex class is O(sqrt(c/k)), and the gap between empirical and theoretical bounds is loose; ImageNet with c=1000 does not show the trend.
Experimental setup
Datasets: CosmosQA Amazon Polarity BoolQ SciQ CIFAR-10 ImageNet
Benchmarks: CosmosQA Amazon Polarity BoolQ SciQ CIFAR-10 ImageNet
Metrics: test cross-entropy loss test accuracy misfit (KL divergence) gain
Baselines: standard cross-entropy weak-to-strong training (single logistic head)
Builds on: weak-to-strong generalization (Charikar et al. squared-loss misfit result) generalized Pythagorean inequality for Bregman divergences Caratheodory's theorem
Closest papers by meaning
- Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.91)
- The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge (NeurIPS 2026 · similarity 0.90)
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.90)
- From Linear to Nonlinear: Provable Weak-to-Strong Generalization through Feature Learning (NeurIPS 2025 · similarity 0.90)
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.90)
- Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.