Research map / Large language models / Alignment and preferences
Aligning Deep Implicit Preferences by Learning to Reason Defensively
Peiming Li, Zhiyuan Hu, Shiyu Li, Xi Chen, Yang Tang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Pers-GenPRM critique-and-process reward instead of scalar reward model (ablation) | GRPO on Qwen2.5-7B-Instruct · DeepPref | Acc_DA: 70.3 -> 76.3 | Improved |
| add critique supervision to the reward model (GRM) (ablation) | GRPO · DeepPref | Acc_DA: 70.3 -> 74.7 | Improved |
| add process supervision to the reward model (PRM) (ablation) | GRPO · DeepPref | Acc_DA: 70.3 -> 73.0 | Improved |
| combine process and critique supervision (Pers-GenPRM) (ablation) | GRPO · DeepPref | Acc_DA: 73.0 -> 76.3 | Improved |
| use Pers-GenPRM reward instead of rubric-based RM (ablation) | GRPO · DeepPref | Innovative Expansion: 34.7 -> 42.7 | Improved |
| use CDRA instead of GRPO with outcome RM | Qwen2.5-7B-Instruct · DeepPref | Acc_Mis: 30.7 -> 32.3 | Mixed |
| use CDRA instead of GRPO | Qwen2.5-7B-Instruct · DeepPref | Acc_PF: 83.7 -> 84.7 | Improved |
| use CDRA instead of SFT | Qwen2.5-7B-Instruct · PrefEval | Thoughtfulness m_th: 21.4 -> 27.7 | Mixed |
| use CDRA instead of CoT | Qwen2.5-7B-Instruct · PrefEval | Thoughtfulness m_th: 54.5 -> 27.7 | Hurt |
| use CDRA instead of GRPO | Qwen2.5-7B-Instruct · ALOE human eval | average alignment: 3.18 -> 3.92 | Improved |
| use CDRA instead of TPO | Qwen2.5-7B-Instruct · ALOE human eval | average alignment: 3.86 -> 3.92 | Improved |
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.91)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.89)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.89)
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback (ICML 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
