Research map / Multi-agent systems and games / Cooperative multi-agent RL
UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UNEX-RL, a multi-agent RL framework for multi-stage recommender systems that uses a cascading information chain (CIC) to handle observation dependency and cascading effects from unidirectional execution, plus stopping gradient and category-quantile rescaling for variance reduction.
Problem
Applying reinforcement learning to optimize long-term user rewards across the multi-stage structure (matching, pre-ranking, ranking, re-ranking) of industrial recommender systems, where unidirectional execution breaks the CTDE assumption of simultaneously sampled observations.
Approach
Model each recommendation stage as an agent and train with a cascading information chain that replays the downstream observations and actions from the first-stage observation via target actors and checkpointed information-extraction parameters, so only the first observation must come from the replay buffer; add stopping gradient and category-quantile reward rescaling to reduce variance.
Key results
- Offline, UNEX-RL-CIC reaches 1056.2 s WatchTime and 24.2 session length versus 763.2 s and 18.9 for TD3 and 654.0 s for CEM.
- Online on Kwai, UNEX-RL-CIC gives +0.953% daily WatchTime over CEM and about +0.558% over TD3.
- UNEX-RL-CTDE gives +0.602% daily WatchTime over CEM online, outperforming single-agent DDPG (+0.219%) and TD3 (+0.395%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CIC instead of CTDE training | UNEX-RL · KuaiRand simulator | WatchTime (s): 887.2 -> 1056.2 | Helped |
| Use CIC instead of CTDE training | UNEX-RL · KuaiRand simulator | Session Length: 21.6 -> 24.2 | Helped |
| Add CQR (ablation) | UNEX-RL-CTDE · KuaiRand simulator | WatchTime (s): 782.5 -> 887.2 | Helped |
| Add CQR (ablation) | UNEX-RL-CTDE · KuaiRand simulator | Session Length: 19.7 -> 21.6 | Helped |
| Add CQR (ablation) | UNEX-RL-CIC · KuaiRand simulator | WatchTime (s): 974.3 -> 1056.2 | Helped |
| Add CQR (ablation) | UNEX-RL-CIC · KuaiRand simulator | Session Length: 22.7 -> 24.2 | Helped |
| Add SG (ablation) | UNEX-RL-CIC · KuaiRand simulator | WatchTime (s): 752.1 -> 1056.2 | Helped |
| Add SG (ablation) | UNEX-RL-CIC · KuaiRand simulator | Session Length: 18.4 -> 24.2 | Helped |
| Use CIC instead of CTDE training | UNEX-RL · Kwai online A/B experiment | Daily WatchTime: + 0.602% -> + 0.953% | Helped |
Surprising findings
- Removing stopping gradient from UNEX-RL-CIC drops offline WatchTime to 752.1 s, below the TD3 single-agent baseline.
Limitations
- Offline experiments rely on a user simulator with a quitting mechanism rather than real users.
- Assumes information-extraction parameters P^i are accessible and checkpoints can replay the online process.
- Evaluated on a single short-video platform with a three-stage setup in offline experiments.
Experimental setup
Datasets: KuaiRand
Benchmarks: KuaiRand (offline simulator) Kwai online A/B test
Metrics: WatchTime Session Length daily WatchTime relative improvement session-wise WatchTime relative improvement
Baselines: CEM DDPG TD3 UNEX-RL-CTDE
Builds on: MADDPG DDPG TD3
Closest papers by meaning
- On the Unexpected Effectiveness of Reinforcement Learning for Sequential Recommendation (ICML 2024 · similarity 0.89)
- Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders (NeurIPS 2026 · similarity 0.88)
- Learning to Recommend in Unknown Games (NeurIPS 2026 · similarity 0.87)
- R$^2$ec: Towards Large Recommender Models with Reasoning (NeurIPS 2025 · similarity 0.86)
- Think before Recommendation: Autonomous Reasoning-enhanced Recommender (NeurIPS 2025 · similarity 0.86)
- AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.