Research map / Label-efficient and robust learning / Recognition and long-tail
Theoretical Analysis of Weak-to-Strong Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives expansion-based error bounds explaining pseudolabel correction and coverage expansion in weak-to-strong generalization, with a statistical procedure to check expansion empirically on IMDb with a SentenceBERT linear probe.
Problem
Existing weak supervision theory cannot explain why a strong student trained on incomplete, incorrect pseudolabels from a weaker teacher corrects the teacher's errors (pseudolabel correction) and generalizes to uncovered inputs (coverage expansion).
Approach
Uses expansion properties of the data distribution (neighbors of bad/uncovered points overlap with good points) combined with student robustness on neighborhoods (including average-case robustness and robust expansion) to bound gold error on covered and uncovered sets by weak error, and shows expansion can be estimated from finite data via a neighborhood oracle.
Key results
- On IMDb with a SentenceBERT linear probe, label 0 expansion 0.848 gives bound 0.05 below alpha_0=0.11 and true error 0.04, showing pseudolabel correction.
- For label 1, bound 0.37 vs weak error 0.33 and true error 0.35, so no pseudolabel correction, consistent with theory.
- Uncovered-set true error for label 1 is 0.29 (bound 0.33), far better than random guessing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a SentenceBERT linear classifier on coarse-rule pseudolabels | SentenceBERT · IMDb sentiment, covered set S₀ | true-label error: 0.11 -> 0.04 | Helped |
| Train a SentenceBERT linear classifier on coarse-rule pseudolabels | SentenceBERT · IMDb sentiment, covered set S₁ | true-label error: 0.33 -> 0.35 | Hurt |
| Train on coarse-rule pseudolabels and evaluate beyond the labeled coverage | SentenceBERT linear classifier · IMDb sentiment, uncovered set T₀ | true-label error: 0.37 bound; 0.16 observed error | Helped |
| Train on coarse-rule pseudolabels and evaluate beyond the labeled coverage | SentenceBERT linear classifier · IMDb sentiment, uncovered set T₁ | true-label error: 0.33 bound; 0.29 observed error | Helped |
Surprising findings
- Bag-of-words student minimizing weak error can exactly reproduce pseudolabels with no weak-to-strong effect, so the student hypothesis class and training details matter.
Limitations
- Expansion checking relies on a heuristic for the worst-expanding set, not an exact computation (coNP-complete in general).
- Experiments are limited in scope: one task, one student model, and one coarse rule teacher.
- No new weak supervision algorithms are proposed.
Experimental setup
Datasets: IMDb
Benchmarks: IMDb
Metrics: expansion estimate error on weak labels error on true labels error bound value
Baselines: Cohen et al. style expansion bound (self-training / adversarially robust bound) trivial triangle-inequality bound co-training bound (Balcan et al.)
Builds on: co-training theory self-training expansion guarantees F-Delta-F distance domain adaptation robust expansion
Closest papers by meaning
- Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.93)
- Weak-to-Strong Generalization Even in Random Feature Networks, Provably (ICML 2025 · similarity 0.92)
- On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspective (NeurIPS 2025 · similarity 0.92)
- Provable weak-to-strong generalization via benign overfitting (ICLR 2025 · similarity 0.91)
- Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimension (ICML 2025 · similarity 0.89)
- Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.