Research map / Large language models / Alignment and preferences
ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
Xiaoqiang Lin, Arun Verma, Zhongxiang Dai, Daniela Rus, See-Kiong Ng, Bryan Kian Hsiang Low
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select preference data with gradient-based uncertainty (ActiveDPO) instead of random/APO/APLP | DPO (Llama-2-7B, Gemma-2B, Qwen3-4B) · TLDR and WebGPT | average reward | Improved |
| Use heuristic APLP selection on Llama-2 | DPO · TLDR and WebGPT, Llama-2-7B | average reward | Hurt |
| Use APLP selection on Gemma | DPO · Gemma-2B | average reward | Improved |
| Use APO (linear reward assumption) | DPO · TLDR and WebGPT | average reward | Mixed |
| Lower random projection dimension below 8192 (ablation) | ActiveDPO · alignment reward | average reward | Hurt |
| Normalize LoRA gradients to unit norm (ablation) | ActiveDPO · WebGPT, Gemma-2B | average reward | Improved |
| Normalize LoRA gradients to unit norm (ablation) | ActiveDPO · TLDR | average reward | No clear effect |
| Choose DPO data per model (ablation) | Gemma-2B models 1 and 2 · three DPO subsets | win-rate | Mixed |
Closest papers by meaning
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.92)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.91)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.91)
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
