Research map / Label-efficient and robust learning / Recognition and long-tail
Strong Empowered and Aligned Weak Mastered Annotation for Weak-to-Strong Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEAM, where a strong model generates principle fast-and-frugal trees of knowledge and an aligned weak model makes the final preference annotation, improving weak-to-strong generalization on six preference datasets.
Problem
Weak-to-strong generalization for preference annotation is limited by the weak model's knowledge scope, and letting the unaligned strong model correct weak annotations is risky for alignment.
Approach
The strong model empowers annotation by generating sample-specific principle fast-and-frugal trees via a searching-while-thinking process with entropy-based information scores, while the aligned weak model keeps final annotation authority by picking informative nodes.
Key results
- SEAM achieves average W2SG accuracy 60.1% and PGR 42%, versus 57.7% (24% PGR) for the best baseline.
- On AHarmless and SafeRLHF, SEAM reaches PGR of 29% and 10%, where all baselines fail to show W2SG.
- Weak annotation accuracy averages 66.3% for SEAM with filtering, a 6.5% improvement over the best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEAM instead of the strongest baseline, Self-Reward | Qwen2-7B W2SG fine-tuning · AHarmless | accuracy: 42.4 -> 50.2 | Helped |
| Use SEAM instead of the strongest baseline, WSC Filter | Qwen2-7B W2SG fine-tuning · SafeRLHF | accuracy: 53.9 -> 54.6 | Helped |
| Use SEAM instead of the strongest baseline, Consultancy | Qwen2-7B W2SG fine-tuning · CaiHarmless | accuracy: 56.6 -> 52.3 | Hurt |
| Use SEAM instead of the strongest baseline, Self-Reward | Qwen2-7B W2SG fine-tuning · HelpSteer | accuracy: 78.1 -> 77.3 | Hurt |
| Add tree information score-based filtering (ablation) | SEAM · AF, HS, AM, CH, AHH, SR | weak annotation accuracy (Avg.): 60.7 -> 66.3 | Helped |
| Add information-score backtracking (ablation) | SEAM · AF, HS, AM, CH, AHH, SR | weak annotation accuracy (Avg.): 60.4 -> 66.3 | Helped |
| Add principle-aware thoughts (ablation) | SEAM · AF, HS, AM, CH, AHH, SR | weak annotation accuracy (Avg.): 58.0 -> 66.3 | Helped |
| Add principle FF-Tree generation (ablation) | SEAM · AF, HS, AM, CH, AHH, SR | weak annotation accuracy (Avg.): 51.7 -> 66.3 | Helped |
| Use SEAM instead of the strongest baseline, WSC Filter | weak annotation generation · AF | weak annotation accuracy: 69.7 -> 67.2 | Hurt |
| Use SEAM instead of the strongest baseline, WSC Filter | weak annotation generation · CH | weak annotation accuracy: 56.4 -> 68.2 | Helped |
Surprising findings
- Strong-model confidence-based correction methods (AuxConf, WSC Filter) yield very low annotation quality on harmlessness datasets, showing risky corrections from an unaligned strong model.
- Multi-agent Debate performs worst among methods, with negative average PGR (-13%).
Limitations
- Dataset-level filtering requires the full held-out distribution to be annotated, which may not hold for streaming annotation (hence the w/o Filter variant).
- Only one weak-strong model pair is main-reported; other pairs are in the appendix.
- Uses 11 predefined principles that are fixed by the authors.
Experimental setup
Datasets: AHelpful HelpSteer AHarmless CaiHarmless AnthropicHH SafeRLHF Alpaca-52K
Benchmarks: AHelpful HelpSteer AHarmless CaiHarmless AnthropicHH SafeRLHF
Metrics: accuracy performance gap recovered (PGR) weak annotation accuracy win/tie/lose rate
Baselines: Naive W2S Uncertain Filter Self-Reward WS-Ensemble AuxConf WSC Filter Consultancy Debate
Builds on: weak-to-strong generalization (Burns et al. 2024) fast-and-frugal trees Direct Preference Optimization Zhao et al. 2023 preference-as-instruction formatting
Closest papers by meaning
- Debate Helps Weak-to-Strong Generalization (AAAI 2025 · similarity 0.92)
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (ICML 2024 · similarity 0.92)
- Weak-to-Strong Generalization under Distribution Shifts (NeurIPS 2025 · similarity 0.91)
- Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions (ICML 2025 · similarity 0.91)
- Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization (ICLR 2025 · similarity 0.91)
- The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.