Research map / Reinforcement learning / Visual and representation RL
On the Unexpected Effectiveness of Reinforcement Learning for Sequential Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RL-based sequential recommenders (SQN) gain on next-item prediction mainly because the RL auxiliary loss makes embeddings encode the user's interaction history length, and replacing RL with a direct history-length regression loss (hist) yields comparable gains.
Problem
RL-based sequential recommenders report state-of-the-art next-item prediction (NIP) performance even though NIP is a short-term, one-shot metric that RL's long-term objective should not optimize; the reason for this gain is unexplained.
Approach
Formally shows an optimal RL policy can be arbitrarily worse than a cross-entropy model under consistent NIP metrics, then diagnoses SQN's gains via feature analysis of Q-values and shows a simple auxiliary loss predicting the number of past interactions (hist) recovers comparable gains without RL.
Key results
- On RetailRocket buy interactions, SASRec-EVAL/NIN-EVAL-type RL variants and hist auxiliary models match or beat SQN; e.g., NIN-SQN reaches HR@5 0.3307 vs NIN 0.2282, and NIN-HIST reaches 0.3001.
- On RC15, CASER-HIST attains the best HR@5 among reported models at 0.4528 vs CASER 0.4095 and CASER-SQN 0.4263.
- Feature analysis on RetailRocket ranks number of past interactions (hist-length) as the most important feature for explaining SQN Q-values (SHAP 18.50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the DQN loss with policy evaluation loss (ablation) | NextItNet · RetailRocket buy interactions | HR@5: 0.2282 -> 0.3308 | Helped |
| Replace the DQN loss with policy evaluation loss (ablation) | NextItNet · RetailRocket buy interactions | NDCG@20: 0.2053 -> 0.2905 | Helped |
| Replace the RL head with prediction of past interaction count (ablation) | Caser · RC15 buy interactions | HR@20: 0.6192 -> 0.6686 | Helped |
| Replace the RL head with prediction of past interaction count (ablation) | GRU · RetailRocket buy interactions | HR@5: 0.1601 -> 0.1980 | Helped |
| Replace the RL head with future interaction-count prediction (ablation) | GRU · RetailRocket buy interactions | HR@5: 0.1601 -> 0.0390 | Hurt |
| Replace the RL head with prediction of the next two interaction types (ablation) | GRU · RetailRocket buy interactions | HR@5: 0.1601 -> 0.1644 | No clear effect |
| Replace the RL head with prediction of interaction types for the next three items (ablation) | GRU · RetailRocket buy interactions | HR@5: 0.1601 -> 0.1696 | Helped |
| Add the DQN loss as an auxiliary loss (ablation) | GRU · RetailRocket buy interactions | HR@5: 0.1601 -> 0.1921 | Helped |
| Add the DQN loss as an auxiliary loss (ablation) | Caser · RC15 buy interactions | HR@5: 0.4095 -> 0.4263 | Helped |
| Replace the DQN loss with policy evaluation loss (ablation) | Caser · RC15 buy interactions | HR@5: 0.4095 -> 0.4340 | Helped |
| Replace the RL head with interaction-type prediction (ablation) | Caser · RC15 buy interactions | HR@5: 0.4095 -> 0.4351 | Helped |
| Replace the RL head with future interaction-count prediction (ablation) | the sequential recommendation models · RetailRocket buy interactions | HR@5: 0.1601 -> 0.0390 | Hurt |
Surprising findings
- Replacing the RL loss with a supervised history-length regression achieves performance comparable to RL-based SQN.
- Predicting the number of future interactions (fut) performs very poorly despite fut-length being the second most important feature.
Limitations
- Analysis limited to SQN-style RL methods evaluated with the NIP protocol; does not assess long-term benefits of RL, which would need a simulator.
- Gap between hist/cat auxiliary losses and full SQN is not entirely closed; hist performance varies across backbones.
- The fut auxiliary loss has high variance and performs poorly.
Experimental setup
Datasets: RetailRocket RC15
Benchmarks: RetailRocket RC15
Metrics: HR@5 NDCG@5 HR@20 NDCG@20
Baselines: GRU NextItNet Caser SASRec SQN GRU-CAT
Builds on: SQN (Self-Supervised Q-learning) DQN GRU4Rec SASRec Caser
Closest papers by meaning
- UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution (AAAI 2024 · similarity 0.89)
- Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders (NeurIPS 2026 · similarity 0.89)
- Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.88)
- ConvRec-R1: Training LLM-based Conversational Recommender Systems with Reinforcement Learning (ICLR 2026 · similarity 0.87)
- R$^2$ec: Towards Large Recommender Models with Reasoning (NeurIPS 2025 · similarity 0.87)
- Think before Recommendation: Autonomous Reasoning-enhanced Recommender (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.