Research map / Large language models / Alignment and preferences
On Softmax Direct Preference Optimization for Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S-DPO, a softmax-based extension of DPO that uses multiple sampled negatives to train LM-based recommenders for personalized ranking, outperforming SFT-based and other LM-based recommenders on three datasets.
Problem
LM-based recommenders are fine-tuned with a language modeling loss on positive items only, which ignores negatives and is not optimized for personalized ranking.
Approach
Extend DPO from pairwise Bradley-Terry to a Plackett-Luce-style partial-ranking loss over one positive and multiple sampled negatives, which connects to a softmax (InfoNCE-like) ranking loss and implicitly weights hard negatives.
Key results
- On Goodreads, S-DPO reaches HR@1 0.6609 vs. 0.5292 for the second-best LLaRA; on LastFM 0.6628 vs. 0.4508.
- S-DPO improves HR@1 by 11.10% to 47.03% over the second-best baseline across datasets, with valid ratio near 0.99.
- With Mistral-7B on LastFM, the language modeling loss baseline (0.7828) slightly exceeds S-DPO with 8 negatives (0.7820).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace LLaRA with S-DPO preference alignment | LLaMA2-7B recommender · Goodreads | HR@1: 0.5292 -> 0.6609 | Helped |
| replace LLaRA with S-DPO preference alignment | LLaMA2-7B recommender · LastFM | HR@1: 0.4508 -> 0.6628 | Helped |
| replace LLaRA with S-DPO preference alignment | LLaMA2-7B recommender · MovieLens | HR@1: 0.4737 -> 0.5263 | Helped |
| add explicit ranking optimization to SFT (ablation) | LLaMA2-7B recommender · Goodreads, LastFM, and MovieLens | HR@1 | Helped |
| add multiple negative samples to pairwise DPO (ablation) | LLaMA2-7B recommender · Goodreads, LastFM, and MovieLens | HR@1 | Helped |
| use K negatives instead of one with DPO (ablation) | LLaMA2-7B recommender · LastFM | HitRatio@1: 0.6342 -> 0.6413 | Helped |
| use K negatives instead of one with DPO (ablation) | LLaMA2-7B recommender · MovieLens | HitRatio@1: 0.4947 -> 0.4947 | No clear effect |
| use K negatives instead of one with DPO (ablation) | LLaMA2-7B recommender · Goodreads | HitRatio@1: 0.6381 -> 0.6628 | Helped |
| replace DPO with S-DPO while using K negatives (ablation) | LLaMA2-7B recommender · LastFM | HitRatio@1: 0.6413 -> 0.6477 | Helped |
| replace DPO with S-DPO while using K negatives (ablation) | LLaMA2-7B recommender · MovieLens | HitRatio@1: 0.4947 -> 0.5263 | Helped |
| replace DPO with S-DPO while using K negatives (ablation) | LLaMA2-7B recommender · Goodreads | HitRatio@1: 0.6628 -> 0.6661 | Helped |
| increase the number of negatives from 1 to 15 (ablation) | S-DPO · Recommendation datasets | HR@1 | Helped |
Surprising findings
- With Mistral-7B on LastFM, plain language modeling loss (HR@1 0.7828) is marginally higher than S-DPO with 8 negatives (0.7820).
Limitations
- Number of negatives capped at 15 due to compute and time constraints.
- Increasing negatives raises training cost, more pronounced for LMs.
- Evaluation uses a 20-item candidate re-ranking setup with HR@1 only, not full ranking metrics like NDCG.
Experimental setup
Datasets: MovieLens Goodreads LastFM
Benchmarks: MovieLens Goodreads LastFM
Metrics: HR@1 Valid Ratio
Baselines: GRU4Rec Caser SASRec MoRec LLaMA2 Chat-REC TALLRec LLaRA
Builds on: DPO Plackett-Luce model BPR TALLRec LLaRA
Closest papers by meaning
- Mult-DPO: Multinomial Direct Preference Optimization for Recommender Systems (NeurIPS 2026 · similarity 0.92)
- Preference Diffusion for Recommendation (ICLR 2025 · similarity 0.90)
- Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.88)
- Advancing Loss Functions in Recommender Systems: A Comparative Study with a Rényi Divergence-Based Solution (AAAI 2025 · similarity 0.88)
- ConvRec-R1: Training LLM-based Conversational Recommender Systems with Reinforcement Learning (ICLR 2026 · similarity 0.88)
- PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.