Research map / Large language models / Alignment and preferences
When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
Amirabbas Afzali, Myeongho Jeon, Maria Brbic
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reweight DPO samples by weak-LLM confidence (CW-DPO) instead of WS-DPO (ablation) | DPO with weak labels · avg of HH-RLHF/TL;DR/UFB, OPT-13B | GRA (%): 57.9 -> 60.3 | Improved |
| reweight IPO samples by confidence (CW-IPO) (ablation) | IPO with weak labels · avg of 3 datasets, OPT-13B | GRA (%): 57.9 -> 61.5 | Improved |
| reweight rDPO samples by confidence (CW-rDPO) (ablation) | rDPO with weak labels · avg of 3 datasets, OPT-13B | GRA (%): 55.5 -> 62.7 | Improved |
| confidence weighting on Qwen2.5-14B strong model (ablation) | DPO/IPO/rDPO · avg of 3 datasets, Qwen2.5-14B | GRA (%): 74.8 -> 75.6 (DPO), 73.7 -> 77.2 (IPO), 74.6 -> 77.3 (rDPO) | Improved |
| confidence weighting on DPO, HH-RLHF with Qwen 14B (ablation) | CW-DPO · HH-RLHF, Qwen2.5-14B | GRA (%): 81.4 -> 80.6 | Hurt |
| use CW-DPO with 30% human labels instead of DPO on 100% human labels | CW-DPO · HH-RLHF, OPT-1.3B | GRA (%): 71.9 -> 69.9 | Hurt |
| use CW-DPO with 30% human labels instead of DPO on 100% human labels | CW-DPO · avg of 4 datasets, Qwen2.5-7B | GRA (%): 65.1 -> 70.6 | Improved |
| use CW-DPO with 20% annotations vs DPO on 100% human labels | CW-DPO · HH-RLHF, OPT-1.3B | GRA (%): 69.7 -> 70.3 | Improved |
| use larger strong models | CW-DPO · HH-RLHF | GRA gain over baselines | Mixed |
| use C1 weighting instead of sigmoid or clipped-margin weights (ablation) | CW-DPO · avg of Harmless/Helpful/HH-RLHF, Qwen | GRA (%): 69.4 -> 72.7 vs C2; 68.4 vs C3; 70.1 vs C4 | Improved |
| train weak annotator with Bradley-Terry scalar head instead of DPO or SFT+DPO (ablation) | weak annotator · unlabeled HH data | reward accuracy (%): 53.8 (DPO), 54.9 (SFT+DPO) -> 64.8 (BT) | Improved |
| use confidence weighting instead of top-N% confidence filtering (ablation) | CW-DPO · HH-RLHF | GRA | Improved |
Closest papers by meaning
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.93)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.93)
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment (NeurIPS 2025 · similarity 0.93)
- Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model (ICLR 2025 · similarity 0.93)
- Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.92)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
