Research map / Large language models / Alignment and preferences

Aligning Language Models with Demonstrated Feedback

ICLR 2025 · Large language models › Alignment and preferences · approach: Steering and correcting LLM behavior · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DITTO, which turns a handful (<10) of user demonstrations into online preference comparisons (demonstrations preferred over the LLM's own and earlier-checkpoint samples) and trains with DPO, outperforming SFT, SPIN, and prompting on author-style alignment and a user study.

Problem

Aligning LLMs to a specific individual or task from only a handful of demonstrations, where standard SFT, RLHF, and prompting need far more data or are brittle.

Approach

Treat expert demonstrations as preferred over samples from the current LM and its earlier iterates, building replay and intermodel comparison rankings online, then optimize with DPO without updating the reference policy; this is derived as online imitation learning with a KL-constrained objective.

Key results

• DITTO averages 71.67% win rate on CMCC and 82.50% on CCAT against GPT-4 judge, versus 56.78% and 73.89% for SFT with Mistral.
• In the user study (N=16), DITTO reaches 68.8% human win rate versus 55.5% for SFT, 51.6% for few-shot GPT-4, and 27.3% for zero-shot GPT-4.
• Demonstration-based DITTO with 4 demos matches DPO on 500 pairwise preferences sampled from the base model, showing demonstrations are more sample-efficient.

Measured effects reported in the paper

Surprising findings

• Few-shot prompting with more demonstrations did not improve over zero-shot-like performance and remained well below DITTO, suggesting RLHF priors are strong.
• Updating the reference policy at each iteration (as in SPIN) substantially degraded win rate (70.1% to 45.8%).

Limitations

• DITTO is slower than prompting and SFT because sampling is a bottleneck.
• DITTO-adapted models forget general capabilities (HumanEval pass@1 drops from 0.31 to 0.13 without adapter routing).
• Only tested on Mistral 7B and not across model families or sizes.

Experimental setup

Datasets: CMCC CCAT HumanEval

Benchmarks: CMCC CCAT50 user study (email tasks) HumanEval

Metrics: win rate pass@1

Baselines: zero-shot prompting few-shot prompting SFT SPIN self-prompt GPT-4 zero-shot GPT-4 few-shot

Builds on: DPO SPIN LoRA Maximum entropy IRL

Closest papers by meaning

• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.90)
• Your Weak LLM is Secretly a Strong Teacher for Alignment (ICLR 2025 · similarity 0.89)
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment (ICLR 2024 · similarity 0.89)
• Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.88)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.88)
• Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.88)