Research map / Label-efficient and robust learning / Recognition and long-tail
MTRec: Learning to Align with User Preferences via Mental Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTRec, which learns a distributional "mental reward" model from implicit feedback via quantile-regression inverse Q-learning and uses it as an auxiliary signal to align sequential recommenders with users' real preferences.
Problem
Recommendation models trained on implicit feedback (clicks) can be misaligned with users' real preferences, since clicks do not always reflect satisfaction.
Approach
Model users as agents maximizing accumulated latent "mental rewards" and recover a stochastic (distributional) reward function from behavior data with QR-IQL, a quantile-regression variant of inverse soft Q-learning; the learned reward is then added as an alignment loss or RL reward term to existing recommenders.
Key results
- On Amazon Electronics, MTRec improved SASRec NCIS from 0.8243 to 0.8833 and DIN NCIS from 0.6044 to 0.8728 (AUC gains are small, often under 0.01).
- In Virtual Taobao, average CTR rose from 0.5435 to 0.678 for PPO and from 0.7055 to 0.909 for SAC.
- Online A/B test on an industrial short-video platform showed about 7% increase in average video viewing time over 7 days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MTRec mental-reward alignment to LinRec | LinRec · Amazon Electronics | AUC: 0.8579 -> 0.8594 | Helped |
| Add MTRec mental-reward alignment to LinRec | LinRec · Amazon Electronics | NCIS: 0.8077 -> 0.9782 | Helped |
| Add MTRec mental-reward alignment to SIGMA | SIGMA · Amazon Books | AUC: 0.8762 -> 0.8814 | Helped |
| Add MTRec mental-reward alignment to LinRec | LinRec · Amazon Books | NCIS: 2.4653 -> 3.7928 | Helped |
| Add predicted mental rewards to PPO training rewards | PPO · Virtual Taobao | average CTR: 0.5435 -> 0.678 | Helped |
| Add predicted mental rewards to SAC training rewards | SAC · Virtual Taobao | average CTR: 0.7055 -> 0.909 | Helped |
| Add MTRec alignment loss to DCN | DCN · Industrial short-video platform, 7-day A/B test | average video viewing time: +7\% | Helped |
| Use distributional QR-IQL instead of non-distributional IQ-Learn for the mental reward model | MTRec recommendation models · Amazon Books and Electronics | AUC | Mixed |
Surprising findings
- Plain IQ-Learn (IRL) alone gives gains nearly as large as the distributional QR-IQL on several models, and on some settings (e.g., SASRec on Electronics AUC, LinRec NCIS on Books) IRL is better than MTRec.
Limitations
- No systematic way to directly evaluate the learned mental reward model; no ground-truth mental reward labels.
- Plans for large-scale human studies are left for future work.
- Mental reward is a latent quantity inferred from implicit behavior, so alignment depends on the assumption that users maximize accumulated mental rewards.
Experimental setup
Datasets: Amazon Books Amazon Electronics Virtual Taobao
Benchmarks: Amazon Books Amazon Electronics Virtual Taobao
Metrics: AUC NCIS episodic CTR (eCTR) average viewing time (online A/B)
Baselines: Wide&Deep PNN DeepFM SASRec DIN DIEN LinRec SIGMA IQ-Learn (IRL) PPO SAC DCN
Builds on: IQ-Learn QR-DQN maximum entropy IRL SASRec RLHF
Closest papers by meaning
- Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders (NeurIPS 2026 · similarity 0.86)
- On the Unexpected Effectiveness of Reinforcement Learning for Sequential Recommendation (ICML 2024 · similarity 0.86)
- Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.86)
- LLM-Powered User Simulator for Recommender System (AAAI 2025 · similarity 0.86)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.85)
- Think before Recommendation: Autonomous Reasoning-enhanced Recommender (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.