Research map / Large language models / Reasoning and chain of thought

ConvRec-R1: Training LLM-based Conversational Recommender Systems with Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ConvRec-R1, a two-stage SFT-then-RL framework for LLM conversational recommenders, with a Remap-Reflect-Adjust distillation pipeline for demonstrations and Rank-GRPO, a rank-level variant of GRPO, evaluated on Reddit-v2.

Problem

LLM-based conversational recommenders generate out-of-catalog items, violate output formats, and degrade toward the end of ranked lists; standard GRPO applies sequence-level rewards with token-level updates, which misaligns with rank-structured outputs.

Approach

Treat each rank as the action unit: assign rank-specific advantages using causal (suffix) DCG returns, and use a rank-level importance ratio given by the geometric mean of token probabilities, combined with an SFT warm start built from teacher LLM outputs refined by remap, reflect, and adjust steps.

Key results

• Llama-3.2-3B with SFT+Rank-GRPO (exp_inf) reaches R@20 0.2368 and N@20 0.1283 on Reddit-v2, surpassing GPT-4o zero-shot (R@20 0.2147, N@20 0.1197) and CRAG (0.2212, 0.1227).
• Qwen2.5-0.5B with SFT+Rank-GRPO (exp_inf) reaches N@20 0.1079 versus 0.0945 for SFT+vanilla GRPO, and outperforms GPT-4o-mini (N@20 0.0973).
• NDCG@20 rises over 30x for Qwen2.5-0.5B after SFT relative to zero-shot (paper's reported improvement).

Surprising findings

• Removing SFT (R1-zero) drops Qwen2.5-0.5B NDCG@20 from 0.1079 to 0.0431, showing SFT warm start is crucial.
• Rank-GRPO (exp_inf) at lower ranks shows reward rising then falling during training, suggesting a retrieve-then-rerank behavior.

Limitations

• Remap-Reflect-Adjust pipeline is designed specifically for movie recommendation; generalization to other item types is only assumed.
• Rank-GRPO reward shaping is simple (DCG-based with penalties); more sophisticated horizon-based shaping is left to future work.
• Off-policy results lag on-policy due to higher variance from importance weighting.

Experimental setup

Datasets: Reddit-v2 Redial

Benchmarks: Reddit-v2

Metrics: Recall@5 Recall@10 Recall@15 Recall@20 NDCG@5 NDCG@10 NDCG@15 NDCG@20

Baselines: Redial KBRD KGSF UniCRS NBCRS GPT-4o-mini (zero-shot) GPT-4o (zero-shot) CRAG Vanilla GRPO GSPO

Builds on: GRPO GSPO DCG InstructGPT

Closest papers by meaning

• Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders (NeurIPS 2026 · similarity 0.91)
• Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.90)
• Reinforced Latent Reasoning for LLM-based Recommendation (ICLR 2026 · similarity 0.89)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.89)
• R$^2$ec: Towards Large Recommender Models with Reasoning (NeurIPS 2025 · similarity 0.88)
• LLM4RSR: Large Language Models as Data Correctors for Robust Sequential Recommendation (AAAI 2025 · similarity 0.88)