Research map / Label-efficient and robust learning / Knowledge distillation
Weak-to-Strong Generalization via Bregman Bias–Variance Decomposition
Gengze Xu, Wei Yao, Ziqiao Wang, Yong Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| switch CE to confidence-adaptive CE (CACE) | W2S training · HH-RLHF, Qwen-1.8B -> Qwen-14B | test accuracy (%): 66.03 -> 69.20 | Improved |
| switch CE to CACE | W2S training · HH-RLHF, Qwen-7B -> Qwen-14B | test accuracy (%): 67.90 -> 69.78 | Improved |
| switch CE to CACE | W2S training · CAI-Harmless, GPT2 -> GPT2-Medium | test accuracy (%): 93.05 -> 93.44 | Improved |
| switch CE to CACE | W2S training · CAI-Harmless, Qwen-1.8B -> Qwen-14B | test accuracy (%): 96.03 -> 96.08 | No clear effect |
| switch CE to RCE | W2S training · CAI-Harmless, Qwen-1.8B -> Qwen-14B | test accuracy (%): 96.03 -> 94.93 | Hurt |
| switch CE to RCE | W2S training · HH-RLHF, Qwen-7B -> Qwen-14B | test accuracy (%): 67.90 -> 69.13 | Improved |
| switch CE to RCE | W2S training · CAI-Harmless, GPT2 -> GPT2-Large | test accuracy (%): 93.88 -> 94.55 | Improved |
| switch CE to RCE under heavily smoothed (near random) teacher labels, alpha=0.001 | W2S training · Amazon Polarity label smoothing | accuracy | Improved |
| average multiple weak teachers (ensemble) instead of a single teacher | W2S training · Amazon Polarity | CE test loss | Improved |
| scale up the strong student capacity | W2S training · Amazon Polarity, GPT2 and Qwen families | bias and variance of CE loss | Improved |
Closest papers by meaning
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.92)
- Relating Misfit to Gain in Weak-to-Strong Generalization Beyond the Squared Loss (ICML 2025 · similarity 0.92)
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.91)
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.90)
- Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.90)
- Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
