Research map / Large language models / Alignment and preferences

Mult-DPO: Multinomial Direct Preference Optimization for Recommender Systems

NeurIPS 2026 · Large language models › Alignment and preferences · approach: DPO design choices · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mult-DPO, a DPO objective for set-wise multi-positive preferences in LLM recommenders, using a multinomial surrogate of the marginalized Plackett-Luce likelihood that upper-bounds the PL DPO loss, plus a multi-level extension Mult²-DPO.

Problem

Vanilla DPO assumes pairwise preferences, but recommender feedback gives set-wise preferences with multiple positives and negatives; exact marginalized Plackett-Luce likelihood over these sets is combinatorial in complexity.

Approach

Replace the intractable marginalized PL event likelihood with a multinomial (IID draw) surrogate defined on the same reward-induced weights, giving a closed-form softmax-style DPO loss that provably upper-bounds the PL-DPO loss, with a sequential multinomial extension for multi-level preferences.

Key results

• On Goodreads with Qwen2.5-3B-Instruct, Mult-DPO reaches NDCG@5 0.1288 vs 0.1252 for LiPO (BT) and 0.1254 for D^3.
• On Reddit-V2 with Qwen2.5-3B-Instruct, Mult-DPO reaches NDCG@5 0.1369 vs 0.1147 for LiPO (BT), the largest margin among datasets.
• On MovieLens-10M with Qwen2.5-0.5B-Instruct, the multi-level Mult2-DPO reaches NDCG@5 0.0732 vs 0.0650 for binary Mult-DPO (about 12% gain).

Surprising findings

• On MovieLens-10M, BIGRec (0.0657) slightly beats Mult-DPO (0.0650) at NDCG@5 for the 0.5B backbone.

Limitations

• The multinomial surrogate is not a ranking distribution and assigns mass to duplicate draw sequences outside the valid ranking space.
• The bound's tightness depends on the relative policy-induced weight of positives vs negatives, which can be loose when positives dominate.
• Experiments are limited to Qwen2.5 backbones up to 7B and to three datasets; the 7B experiment reuses the beta selected at smaller scales.

Experimental setup

Datasets: MovieLens-10M Goodreads Reddit-V2

Benchmarks: Goodreads MovieLens-10M Reddit-V2

Metrics: NDCG@5 NDCG@15 NDCG@20

Baselines: BIGRec D^3 Vanilla DPO DMPO S-DPO LiPO (BT)

Builds on: DPO Plackett-Luce model S-DPO SPRec LiPO

Closest papers by meaning

• On Softmax Direct Preference Optimization for Recommendation (NeurIPS 2024 · similarity 0.92)
• Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.89)
• MallowsPO: Fine-Tune Your LLM with Preference Dispersions (ICLR 2025 · similarity 0.89)
• Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.89)
• TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights (ICLR 2025 · similarity 0.89)
• 3D-Properties: Identifying Challenges in DPO and Charting a Path Forward (ICLR 2025 · similarity 0.88)