Research map / Learning theory and algorithms / Prediction and decision losses

Model Alignment as Prospect Theoretic Optimization

ICML 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Losses for decision-making · 23 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Frames LLM alignment objectives as prospect-theoretic human-aware losses (HALOs) and proposes KTO, which aligns models from binary desirable/undesirable signals and matches or beats DPO from 1B to 30B.

Problem

Alignment methods like RLHF and DPO require preference data, and it is unclear why they work so well or whether feedback must be in preference form. The paper asks what inductive biases make alignment objectives succeed.

Approach

Defines human-aware losses (HALOs) as objectives whose implied utility is a prospect-theory value function of the implicit reward relative to a reference point, showing DPO and PPO-Clip qualify and CSFT and SLiC do not. Derives KTO from the Kahneman-Tversky value function, which maximizes utility of each output using only a binary desirable/undesirable label.

Key results

• KTO matches or exceeds DPO at 1B to 30B scales in GPT-4-judged winrate against SFT targets; for Llama models KTO alone is significantly better than DPO alone at 7B and 30B.
• On Zephyr-β-SFT aligned with UltraFeedback, KTO improves GSM8K from 40.0 (DPO) to 53.5, a 13.5-point gain.
• KTO retains outperformance with up to 90% of desirable examples discarded on Llama-7B, and with one output per input on OpenAssistant (Mistral-7B winrate 0.631 vs DPO 0.600).

Measured effects reported in the paper

Surprising findings

• Offline PPO with dummy +1/-1 rewards matches DPO at most scales, suggesting preference data may not be needed if the loss has the right inductive bias.
• KTO can match SFT+DPO without an SFT stage at 13B and 30B scale, while DPO without SFT produces rambling, hallucinated multi-turn outputs.

Limitations

• Converting preference pairs into desirable/undesirable labels is a naive assumption; the paper does not explore more complex decompositions.
• The value function is borrowed from monetary gambles and may not reflect how humans perceive text quality.
• Main evaluation relies on GPT-4 judgments, with human evaluation on only 214 comparisons per method.

Experimental setup

Datasets: Anthropic-HH OpenAssistant SHP UltraFeedback MMLU GSM8K HumanEval BigBench-Hard TydiQA AlpacaEval 2

Benchmarks: MMLU GSM8K HumanEval BigBench-Hard TydiQA AlpacaEval 2

Metrics: GPT-4 winrate vs SFT target exact match pass@1 F1 % win

Baselines: DPO CSFT SLiC PPO (offline) ORPO SFT

Builds on: DPO RLHF PPO-Clip prospect theory

Closest papers by meaning

• Learning to Decide with AI Assistance under Human-Alignment (NeurIPS 2026 · similarity 0.88)
• Direct Alignment with Heterogeneous Preferences (NeurIPS 2025 · similarity 0.87)
• Learning Optimal Advantage from Preferences and Mistaking It for Reward (AAAI 2024 · similarity 0.87)
• Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model (ICML 2025 · similarity 0.87)
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.87)
• Enhancing Preference-based Linear Bandits via Human Response Time (NeurIPS 2024 · similarity 0.87)