Research map / Large language models / Alignment and preferences
Aligner, Diagnose Thyself: A Meta-Learning Paradigm for Fusing Intrinsic Feedback in Preference Alignment
Mengyang Li, Pinlong Zhao, Zhong Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add loss and uncertainty diagnostics to PPLDiff-only meta-learner (ablation) | Ours (PPLDiff only) · Golden HH (30% noise) | reward accuracy (%): 95.8 -> 97.1 | Improved |
| add PPLDiff and uncertainty to Loss-only meta-learner (ablation) | Ours (Loss only) · Golden HH (30% noise) | reward accuracy (%): 88.4 -> 97.1 | Improved |
| add PPLDiff and loss to uncertainty-only meta-learner (ablation) | Ours (Uncertainty only) · Golden HH (30% noise) | reward accuracy (%): 84.7 -> 97.1 | Improved |
| use 2-layer MLP instead of 1-layer (ablation) | meta-learner · Golden HH (30% noise) | reward accuracy (%): 94.2% -> 96.0% | Improved |
| use 3 or 4 layers instead of 2 (ablation) | meta-learner · Golden HH (30% noise) | reward accuracy (%): 96.0% -> 96.2% (3-layer), 96.1% (4-layer) | No clear effect |
| vary hidden width 50 to 200 (ablation) | meta-learner · Golden HH (30% noise) | reward accuracy (%): 95.5% -> 96.1% | No clear effect |
| use 2-layer MLP instead of linear mapping (ablation) | meta-learner · Golden HH (30% noise) | reward accuracy (%): 91.5% -> 96.0% | Improved |
| use attention or gated fusion instead of MLP (ablation) | meta-learner · Golden HH (30% noise) | reward accuracy (%): 96.0% -> 95.8% (attention), 95.6% (gated) | No clear effect |
| add response length ratio as a fourth diagnostic (ablation) | Ours (Fusion) · Golden HH (30% noise) | reward accuracy (%): 96.0% -> 96.1% | No clear effect |
| add embedding distance as a fourth diagnostic (ablation) | Ours (Fusion) · Golden HH (30% noise) | reward accuracy (%): 96.0% -> 96.0% | No clear effect |
| inject label noise into the clean meta-dataset (5%) (ablation) | Ours (Fusion) · Golden HH (30% noise) | reward accuracy (%): 96.0 -> 92.5 | Hurt |
| use expert-curated meta-set instead of random sampling (ablation) | Ours (Fusion) · Golden HH (30% noise) | reward accuracy (%): 96.0% -> 96.8% | Improved |
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.92)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.92)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.91)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
