Research map / Image generation / Text-to-image diffusion
Bayesian WeakS-to-Strong from Text Classification to Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends weak-to-strong generalization to an ensemble of weak models with Bayesian evidential deep learning for soft supervision, and to text generation (slot filling) with token-level soft labels and cDPO using weak-model preferences.
Problem
Weak-to-strong supervision is limited by errors in a single weak model and has mainly been studied for classification, not generation; it also needs to capture variability in human opinions for superalignment.
Approach
Use multiple weak models as observations of human opinion and learn a Dirichlet prior over weak labels via evidential deep learning, with the strong model predicting the Dirichlet parameters; extend to generation via word-mediated token-level soft labels and apply conservative DPO with weak-model-scored preferences.
Key results
- On SciQ classification, Bayesian Multi-Weak with five weak models and auxiliary loss reached average PGR 0.828 and accuracy 0.866, versus 0.754 for Naive Multi-Weak with five weak models.
- On SLURP slot filling, Bayesian Multi-Weak with five weak models reached average PGR 0.668 (SLU-F1 0.721), and cDPO raised average PGR to 0.705.
- Strong ceilings were 0.898 accuracy (Qwen-7B) on SciQ and 0.748 SLU-F1 (Llama-2-7B) on SLURP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Bayesian Multi-Weak instead of Naive Multi-Weak with three teachers | WeakS-to-Strong-3 with Qwen-7B · SciQ | PGR: 0.661 -> 0.756 | Helped |
| Use five rather than three weak models with Bayesian Multi-Weak | WeakS-to-Strong with Qwen-7B · SciQ | PGR: 0.756 -> 0.828 | Helped |
| Add auxiliary loss to Bayesian Multi-Weak (ablation) | WeakS-to-Strong-5 with Qwen-7B · SciQ | accuracy: 0.831 ± 0.008 -> 0.866 ± 0.006 | Helped |
| Add auxiliary loss to Bayesian Multi-Weak (ablation) | WeakS-to-Strong-5 with Llama-2-7B · SLURP test | SLU-F1: 0.718 ± 0.014 -> 0.721 ± 0.013 | Helped |
| Apply cDPO after supervised finetuning | Bayesian Multi-Weak-5 with auxiliary loss · SLURP test | PGR: 0.668 ± 0.166 -> 0.705 ± 0.187 | Helped |
| Estimate target wordpiece probabilities rather than use one-hot sequence labels (ablation) | WeakS-to-Strong sequence generation · SLURP test | SLU-F1: 0.656 ± 0.016 -> 0.704 ± 0.025 | Helped |
| Add probabilities for other categories to target wordpiece probability estimation (ablation) | WeakS-to-Strong sequence generation · SLURP test | PGR: 0.453 ± 0.305 -> 0.668 ± 0.166 | Helped |
| Use fixed performance-based weak-model weights instead of dynamic confidence-based weights (ablation) | WeakS-to-Strong sequence generation · SLURP test | PGR: 0.241 ± 0.163 -> 0.668 ± 0.166 | Helped |
| Add auxiliary loss to Naive Multi-Weak (ablation) | WeakS-to-Strong-5 with Llama-2-7B · SLURP test | SLU-F1: 0.716 ± 0.010 -> 0.694 ± 0.012 | Hurt |
| Include TinyLlama in Joint Decoding scoring (ablation) | Joint Decoding-5 · SLURP test | SLU-F1: 0.706 ± 0.009 -> 0.671 ± 0.010 | Hurt |
| Increase Joint Decoding beam size from M=5 to M=10 (ablation) | Joint Decoding-5 · SLURP test | SLU-F1: 0.671 ± 0.010 -> 0.650 ± 0.004 | Hurt |
| Add auxiliary loss to Bayesian Multi-Weak (ablation) | WeakS-to-Strong-3 with Qwen-7B · CosmosQA | accuracy: 0.694 ± 0.007 -> 0.760 ± 0.005 | Helped |
Surprising findings
- Naive multi-weak ensembles with three weak models did not outperform the best single weak model on classification.
- Adding auxiliary self-training loss was not effective for naive generation ensembles but helped the Bayesian approach.
Limitations
- Evaluated on only two tasks (binary-reformatted SciQ/CosmosQA classification and SLURP slot filling).
- Only three and five weak models were tested due to computational limits.
- Joint Decoding degrades sharply when a poor weak model (TinyLlama) is included.
Experimental setup
Datasets: SciQ SLURP CosmosQA
Benchmarks: SciQ SLURP CosmosQA
Metrics: accuracy SLU-F1 performance gap recovered (PGR)
Baselines: Naive Multi-Weak FlyingSquid Joint Decoding single weak-to-strong models
Builds on: Weak-to-Strong generalization evidential deep learning direct preference optimization conservative DPO FlyingSquid
Closest papers by meaning
- Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model (ICLR 2025 · similarity 0.91)
- Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.91)
- Quantifying the Gain in Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.90)
- Theoretical Analysis of Weak-to-Strong Generalization (NeurIPS 2024 · similarity 0.88)
- MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization (ICLR 2025 · similarity 0.88)
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.