AI Paper Atlas

Research map / Large language models / Alignment and preferences

Capability Traps in DPO

Marco Pollanen

ICML 2026 · Large language models › Alignment and preferences · approach: DPO design choices

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add transient beta=0.02 exposure before constant beta=0.01 (anneal) vs constant 400 steps (ablation)DPO Mistral-7B · Arithmetic probeprobe margin: +1.87 (C minus B), dz = 3.73, p = 0.002Hurt
add transient beta=0.02 exposure before constant beta=0.01 (ablation)DPO Mistral-7B · Format probeprobe margin: +1.08 (C minus B), dz = 3.38, p = 0.003Hurt
add transient high-beta exposure (ablation)DPO Mistral-7B · Logic probeprobe margin: -0.12, p = 0.663No clear effect
add transient high-beta exposure (ablation)DPO Mistral-7B · Sycophancy probeprobe margin: +1.40, p = 0.311No clear effect
raise preference margin via beta (select by margin)DPO LLaMA-2-7B · Logic probesPearson r: r = -0.91Hurt
select beta by high preference marginDPO Mistral-7B 1000 steps · beta=0.010margin vs logic: margin 41.1, logic -0.96Hurt
increase beta toward 0.02DPO Mistral-7B · GSM8K vs logicPearson r: r = -0.89, p = 0.017Mixed
extend training from 200 to 1000 stepsDPO Mistral-7B · beta sweepnon-monotonicityNo clear effect
vary learning rate by 20x (1e-5, 5e-5, 2e-4)DPO LLaMA-2-7B · Logic probelogic margin: negative across all 9 conditions, spread 0.125No clear effect
use Mistral-7B instead of LLaMA-2-7B at beta=0.006DPO · Sycophancy seed variancevariance: 0.0003 -> 0.783Hurt
expand logic suite from 3 to 14 probesDPO Mistral-7B · Logic aggregateaggregate sign: +0.04 -> positive at every betaMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy