Research map / Large language models / Language, safety and interpretability
Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RecBench, a benchmark comparing LLM-based recommenders with conventional DLRMs and sequential recommenders across four item representations (UID, text, semantic embedding, semantic ID), two tasks (CTR and sequential recommendation), five datasets, and up to 17 LLMs, with accuracy and latency measured.
Problem
Existing benchmarks for LLM-as-recommender evaluate only one scenario, narrow item representations, and few models and datasets, and rarely measure efficiency, so LLM and conventional recommenders are hard to compare fairly.
Approach
A unified benchmark that systematically crosses item representation forms (UID, text, semantic embedding, semantic ID) with pair-wise CTR and list-wise sequential recommendation, evaluates zero-shot and fine-tuned LLMs against conventional DLRMs and SeqRec models, and reports inference latency alongside accuracy.
Key results
- Large LLMs (Llama-3 7B, Mistral 7B) fine-tuned with text achieve over 5% AUC gain over the best conventional recommender (DNN_emb) on average (0.7578 vs 0.7171 overall AUC for Mistral-2 7B).
- Conventional DNN_emb retains about 95% of the performance of top LLM recommenders while running thousands of times faster.
- In list-wise recommendation, SID-BERT with conditional beam search reaches overall nDCG@10 of 0.1877 versus 0.0698 for SASRec_24L, while LLM recommenders require roughly 1000x more inference time than SASRec.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Llama-3 7B instead of the best conventional CTR model | Pair-wise recommendation · Goodreads | AUC: 0.5999 -> 0.6826 | Helped |
| Use fine-tuned LLM recommenders rather than conventional DLRMs | Pair-wise recommendation · MIND, MicroLens, Goodreads, CDs, H&M | AUC | Mixed |
| Initialize CTR item representations with Llama-1 semantic embeddings instead of text features | Conventional CTR models · Pair-wise recommendation | AUC: 0.6919 -> 0.7171 | Helped |
| Represent CTR items with text instead of unique IDs | Conventional CTR models · Pair-wise recommendation | AUC: 0.6731 -> 0.6919 | Helped |
| Fine-tune LLMs for recommendation rather than use them zero-shot | LLM recommenders · Pair-wise recommendation | recommendation accuracy: 22\% to 43\% | Helped |
| Use conditional beam search instead of standard beam search (ablation) | SID-SASRec · Overall list-wise recommendation | nDCG@10: 0.0081 -> 0.0306; 0.0109 -> 0.0341; 0.0117 -> 0.0398 | Helped |
| Use conditional beam search instead of standard beam search (ablation) | SID-BERT base · Overall list-wise recommendation | nDCG@10: 0.0941 -> 0.1877 | Helped |
| Use conditional beam search instead of standard beam search (ablation) | SID-Llama-3 7B · Overall list-wise recommendation | nDCG@10: 0.0742 -> 0.1607 | Helped |
| Replace unique item IDs with semantic IDs (ablation) | Fine-tuned BERT list-wise recommender · Overall list-wise recommendation | nDCG@10: 0.1025 -> 0.1877 | Helped |
| Replace unique item IDs with semantic IDs (ablation) | Fine-tuned Llama-3 list-wise recommender · Overall list-wise recommendation | nDCG@10: 0.0257 -> 0.1607 | Helped |
| Increase SASRec depth from 12 to 24 layers (ablation) | SASRec · Overall list-wise recommendation | nDCG@10: 0.0672 -> 0.0698 | Helped |
| Use the more detailed P2 prompt instead of concise P1 (ablation) | Qwen-3 8B zero-shot · MIND, MicroLens, H&M | AUC: 0.5847 -> 0.6036; 0.6543 -> 0.6618; 0.6499 -> 0.6678 | Helped |
Surprising findings
- Fine-tuned UID-based LLMs still lag behind conventional UID-based CTR models, and fine-tuning semantic-ID LLMs performs poorly in CTR.
- Few-shot prompting does not always help and can sharply reduce performance (e.g., on H&M).
Limitations
- Fairness and privacy are not addressed in the main study.
- Models larger than 8B parameters were omitted, and the Qwen3 experiments are limited to an appendix.
- Full metric results are deferred to a webpage, and only AUC and nDCG@10 are shown in the main text.
Experimental setup
Datasets: H&M MIND MicroLens Goodreads Amazon CDs
Benchmarks: H&M MIND MicroLens Goodreads Amazon CDs
Metrics: AUC GAUC nDCG@10 MRR Recall latency (ms)
Baselines: DNN DCNv2 DeepFM AutoInt GDCN SASRec GRU4Rec Caser BERT4Rec P5 RecGPT
Builds on: P5 TIGER LC-Rec SASRec LoRA
Closest papers by meaning
- Improving LLMs for Recommendation with Out-Of-Vocabulary Tokens (ICML 2025 · similarity 0.91)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.91)
- LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation (AAAI 2025 · similarity 0.91)
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.90)
- Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.89)
- LLM4RSR: Large Language Models as Data Correctors for Robust Sequential Recommendation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.