Research map / Large language models / Reasoning and chain of thought

Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shop-R1 applies GRPO-based reinforcement learning with a format reward, self-certainty rationale reward, and a hierarchical, difficulty-aware action reward to simulate human web-shopping behavior, reaching 27.72% exact-action accuracy versus 16.76% for SFT.

Problem

Simulating realistic human online shopping behavior with LLMs: next-step rationale and action prediction on web pages, where SFT on LLM-synthesized rationales is bounded by the rationale generator and sparse binary RL rewards give little guidance.

Approach

Decompose the behavior simulation into rationale generation and action prediction, with a self-certainty (KL-to-uniform) reward for rationales and a hierarchical action reward (action type plus ROUGE-L-based sub-action credit) scaled by a difficulty-aware factor to prevent reward hacking toward trivial terminate actions.

Key results

• Shop-R1 on Qwen-2.5-3B-Instruct reaches 27.72% exact action accuracy vs 16.76% for SFT (about +65% relative).
• Action type accuracy rises to 36.40% and type F1 to 31.28% for the 3B model, versus 22.25% and 24.52% for SFT.
• On OPeRA with Qwen2.5-VL-3B, SFT + RL reaches 38.44% exact action accuracy vs 20.23% for SFT.

Surprising findings

• Binary RL alone from the base model reaches only 1.01% exact action accuracy, and adding binary RL after SFT slightly lowers exact accuracy (16.55%).
• The 0.5B Shop-R1 model reaches 27.72% exact accuracy mainly by over-predicting terminate (97.07% per-type exact accuracy), suggesting shortcut behavior.

Limitations

• Relies on Claude 3.5 Sonnet-generated rationales for SFT cold start and on a proprietary e-commerce dataset (Shop-CART).
• Reward weights (DARS factor 1000, alpha, beta) are set as normalization constants without systematic sensitivity analysis across tasks.
• Smaller models can reach aggregate exact match by over-predicting terminate, indicating reduced behavioral diversity.

Experimental setup

Datasets: Shop-CART OPeRA

Benchmarks: Shop-CART OPeRA

Metrics: exact action accuracy action type accuracy action type F1

Baselines: Zero-shot prompting RL (Binary) SFT SFT + RL (Binary) Claude-3.5-Sonnet

Builds on: GRPO DeepSeek-R1 verl Self-certainty (intrinsic reward)

Closest papers by meaning

• ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents (AAAI 2026 · similarity 0.88)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning (ICLR 2025 · similarity 0.88)
• Reward Reasoning Models (NeurIPS 2025 · similarity 0.87)
• RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.87)
• Boosting LLM Reasoning via Human-Inspired Reward Shaping (NeurIPS 2026 · similarity 0.87)
• ToolRL: Reward is All Tool Learning Needs (NeurIPS 2025 · similarity 0.87)