Research map / Reinforcement learning / RL from feedback

Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that GRPO with binary rewards optimizes AUC, beam-search negatives shift it toward one-way partial AUC, and proposes windowed partial AUC (WPAUC) with a soft threshold-adjusted reweighting (TAWin) for RL-trained LLM recommenders.

Problem

RL training of LLM-based recommenders benefits from beam-search negatives over random negatives, but the mechanism behind this and how it aligns with Top-K recommendation metrics is not well understood.

Approach

Views GRPO with binary reward as pairwise AUC maximization, shows hard beam-search negatives induce OPAUC, and generalizes to a windowed partial AUC over an FPR interval [alpha, alpha+d]; TAWin implements the window via a soft threshold-adjusted Top-K reweighting of negative advantages.

Key results

• TAWin achieves the best R@1, R@3, and N@3 on all four datasets, e.g., Toys R@1 0.0471 vs. 0.0411 for ReRe and 0.0371 for D3.
• Reports average relative improvements of 84.9%, 52.0%, and 5.5% over traditional, generative, and LLM-based recommenders respectively.
• TAWin improves over ReRe across Qwen2.5-3B/7B and Llama-3.2-1B backbones and under GSPO and DAPO optimizers.

Surprising findings

• Random-sampling GRPO is shown to be equivalent to AUC maximization, which the authors argue is misaligned with Top-K metrics.

Limitations

• Theoretical equivalences rely on idealized assumptions (population normalization of advantages, infinite beam width limit for beam search equivalence).
• Experiments are limited to a 0.5B base model in the main setting and to small Amazon and Yelp datasets.

Experimental setup

Datasets: Amazon Toys and Games Amazon Industrial and Scientific Amazon Office Products Yelp

Benchmarks: Amazon Toys Amazon Industrial Amazon Office Yelp

Metrics: Recall@K NDCG@K

Baselines: GRU4Rec Caser SASRec TIGER LC-Rec MiniOneRec BigRec D3 S-DPO ReRe

Builds on: GRPO one-way partial AUC soft top-k operator

Closest papers by meaning

• ConvRec-R1: Training LLM-based Conversational Recommender Systems with Reinforcement Learning (ICLR 2026 · similarity 0.91)
• On the Unexpected Effectiveness of Reinforcement Learning for Sequential Recommendation (ICML 2024 · similarity 0.89)
• UNEX-RL: Reinforcing Long-Term Rewards in Multi-Stage Recommender Systems with UNidirectional EXecution (AAAI 2024 · similarity 0.88)
• LLM-Powered User Simulator for Recommender System (AAAI 2025 · similarity 0.87)
• Think before Recommendation: Autonomous Reasoning-enhanced Recommender (NeurIPS 2025 · similarity 0.87)
• Reinforced Latent Reasoning for LLM-based Recommendation (ICLR 2026 · similarity 0.87)