Research map / Large language models / Alignment and preferences
Capability Traps in DPO
Marco Pollanen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add transient beta=0.02 exposure before constant beta=0.01 (anneal) vs constant 400 steps (ablation) | DPO Mistral-7B · Arithmetic probe | probe margin: +1.87 (C minus B), dz = 3.73, p = 0.002 | Hurt |
| add transient beta=0.02 exposure before constant beta=0.01 (ablation) | DPO Mistral-7B · Format probe | probe margin: +1.08 (C minus B), dz = 3.38, p = 0.003 | Hurt |
| add transient high-beta exposure (ablation) | DPO Mistral-7B · Logic probe | probe margin: -0.12, p = 0.663 | No clear effect |
| add transient high-beta exposure (ablation) | DPO Mistral-7B · Sycophancy probe | probe margin: +1.40, p = 0.311 | No clear effect |
| raise preference margin via beta (select by margin) | DPO LLaMA-2-7B · Logic probes | Pearson r: r = -0.91 | Hurt |
| select beta by high preference margin | DPO Mistral-7B 1000 steps · beta=0.010 | margin vs logic: margin 41.1, logic -0.96 | Hurt |
| increase beta toward 0.02 | DPO Mistral-7B · GSM8K vs logic | Pearson r: r = -0.89, p = 0.017 | Mixed |
| extend training from 200 to 1000 steps | DPO Mistral-7B · beta sweep | non-monotonicity | No clear effect |
| vary learning rate by 20x (1e-5, 5e-5, 2e-4) | DPO LLaMA-2-7B · Logic probe | logic margin: negative across all 9 conditions, spread 0.125 | No clear effect |
| use Mistral-7B instead of LLaMA-2-7B at beta=0.006 | DPO · Sycophancy seed variance | variance: 0.0003 -> 0.783 | Hurt |
| expand logic suite from 3 to 14 probes | DPO Mistral-7B · Logic aggregate | aggregate sign: +0.04 -> positive at every beta | Mixed |
Closest papers by meaning
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss (ICLR 2025 · similarity 0.84)
- Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification (ICML 2025 · similarity 0.84)
- STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability (NeurIPS 2026 · similarity 0.83)
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.83)
- Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.83)
- Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE (ICML 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
