Research map / Large language models / Agents, code and math
AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgentRecBench, a textual interactive simulator over Yelp, Goodreads and Amazon with classic, evolving-interest and cold-start scenarios, and benchmarks 8 LLM agentic recommenders against MF and LightGCN.
Problem
Agentic LLM-based recommender systems lack standardized evaluation protocols, so their designs and performance cannot be systematically compared.
Approach
A unified textual U-R-I interaction environment with standardized query functions and scenario/task-based data visibility control, plus a modular agent framework, used to benchmark agentic recommenders across classic, evolving-interest and cold-start settings with HR@N over 20 candidates.
Key results
- Baseline666 reaches 69.0 average HR@N on Amazon with Qwen-72B in classic setting, versus 15.0 for MF and LightGCN mean-prediction references.
- On Yelp classic, all agentic methods score roughly 3.7 to 7.7 HR, indicating very low absolute accuracy on that dataset.
- Feature-engineered workflows (Baseline666, DummyAgent, RecHackers) substantially outperform BaseAgent, CoTAgent, MemoryAgent and CoTMemAgent across settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use platform-aware feature extraction and a structured agent workflow instead of simpler agent designs | Agentic recommender systems · Classic recommendation across Amazon, Goodreads, and Yelp | average HR@N (N=1,3,5) | Helped |
| Use platform-aware feature extraction and a structured agent workflow instead of simpler agent designs | Agentic recommender systems · Cold-start recommendation | average HR@N (N=1,3,5) | Mixed |
| Evaluate recommendations in cold-start conditions rather than classic conditions | Recommendation methods · Cold-start recommendation across datasets | average HR@N (N=1,3,5) | Hurt |
| Evaluate agents with evolving user interests rather than static classic recommendation | Agentic recommender systems · Evolving-interest recommendation | average HR@N (N=1,3,5) | Mixed |
| Use agent-based methods such as Baseline666 and RecHackers for evolving-interest recommendation | Recommendation methods · Evolving-interest recommendation | average HR@N (N=1,3,5) | Helped |
| Train MF or LightGCN on the sparse interaction data rather than report mean prediction | Traditional and deep learning recommendation baselines · Classic recommendation | average HR@N (N=1,3,5) | No clear effect |
Surprising findings
- Traditional MF and LightGCN could not learn meaningful patterns from the sparse data and were reported only as mean-prediction references.
- MemoryAgent and CoTMemAgent score 0.0 with GPT-4o-mini on Amazon and Goodreads classic tasks, suggesting failure modes with that model.
Limitations
- Traditional and deep learning baselines are limited; the authors plan to add more.
- Environment is text-only and lacks multimodal data.
- Only single-agent systems are evaluated; multi-agent recommenders are not covered.
Experimental setup
Datasets: Yelp Goodreads Amazon
Benchmarks: AgentRecBench classic AgentRecBench cold-start AgentRecBench evolving-interest
Metrics: HR@1 HR@3 HR@5
Baselines: MF LightGCN BaseAgent CoTAgent MemoryAgent CoTMemAgent Baseline666 DummyAgent RecHackers Agent4Rec
Builds on: Agent4Rec chain-of-thought prompting LightGCN Matrix Factorization
Closest papers by meaning
- AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation (ICML 2026 · similarity 0.91)
- Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation (NeurIPS 2025 · similarity 0.88)
- Building Higher-Order Abstractions from the Components of Recommender Systems (AAAI 2024 · similarity 0.88)
- ORBIT - Open Recommendation Benchmark for Reproducible Research with Hidden Tests (NeurIPS 2025 · similarity 0.87)
- LLMRG: Improving Recommendations through Large Language Model Reasoning Graphs (AAAI 2024 · similarity 0.87)
- Re2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.