Research map / Large language models / AI and society
ORBIT - Open Recommendation Benchmark for Reproducible Research with Hidden Tests
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ORBIT, a standardized public recommendation benchmark (5 public datasets, 12 models) plus ClueWeb-Reco, a privacy-preserving hidden webpage-recommendation test set built by soft-matching consented browsing histories to ClueWeb22, and evaluates LLM query generation as a baseline.
Problem
Recommender system research suffers from datasets that do not reflect realistic browsing behavior, raise consent concerns, and inconsistent evaluation settings (splits, candidate pools, metrics), leading to ambiguous and non-reproducible conclusions.
Approach
A unified benchmark with fixed leave-one-out splits and metrics on public datasets, plus a hidden-test webpage recommendation set created by privately mapping consented real browsing histories onto public ClueWeb22 pages via dense retrieval (soft matching), with LLM-generated queries used as a retrieval-based baseline.
Key results
- On the public benchmark, HLLM achieves the highest average NDCG@10 (0.0641), with best results on Toys, Sports and Books, while showing poor performance on Amazon Beauty.
- On the ClueWeb-Reco hidden test, DeepSeek-V3-QueryGen reaches the highest Recall@10 (0.0127) among zero-shot systems, versus 0.0088 for HLLM and 0.0020 for TASTE.
- Soft matching yields 87M-page candidate pool; 100 human-annotated mappings show a positive correlation between retrieval score and relevance, with Cohen's kappa 0.372 (moderate agreement).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use content-based recommendation models instead of ID-based models | Sequential recommendation models · Amazon Books | NDCG@10 | Helped |
| Use HLLM rather than TASTE for content-based recommendation | ClueWeb-Reco zero-shot recommendation · ClueWeb-Reco hidden test | Recall@10: 0.0020 -> 0.0088 | Helped |
| Generate retrieval queries with LLMs rather than use content-based recommendation baselines | ClueWeb-Reco zero-shot recommendation · ClueWeb-Reco hidden test | Recall@10: 0.0088 -> 0.0127 | Helped |
| Use an LLM-QueryGen approach on a large candidate pool | ClueWeb-Reco recommendation · ClueWeb-Reco hidden test | NDCG@10 | Mixed |
| Increase model scale to a billion-scale HLLM architecture | HLLM · Amazon Beauty | NDCG@10 | Hurt |
Surprising findings
- HLLM performs very poorly on Amazon Beauty (NDCG@10 0.0027) despite strong results elsewhere, attributed to small training set size.
- Traditional recommenders (TASTE, HLLM) perform poorly on the large-pool ClueWeb-Reco hidden test, while simple LLM query generation with retrieval is competitive.
Limitations
- Covers a limited set of models and datasets; ML-20M and other datasets were excluded.
- ClueWeb-Reco focuses on U.S. users and is small (1,024 sessions), and the soft-matched data is synthetic rather than the original browsing histories.
- Moderate inter-annotator agreement (kappa 0.372) on relevance of soft-matched pages; no error bars or significance tests reported.
Experimental setup
Datasets: MovieLens-1M Amazon Reviews 2023 (Beauty, Toys, Sports, Books) ClueWeb22-B EN ClueWeb-Reco
Benchmarks: ORBIT public benchmark (ML-1M, Amazon Beauty, Toys, Sports, Books) ClueWeb-Reco hidden test
Metrics: Recall@K NDCG@K
Baselines: GRU4Rec SASRec Caser HGN STAMP FDSA BERT4Rec S3-Rec HSTU SASRecF TASTE HLLM
Builds on: SASRec HSTU HLLM TASTE RecBole
Closest papers by meaning
- RecFlow: An Industrial Full Flow Recommendation Dataset (ICLR 2025 · similarity 0.88)
- AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems (NeurIPS 2025 · similarity 0.87)
- Ready for You When You Are Back: Content-Driven Session-Based Recommendation for Continuity of Experience (AAAI 2025 · similarity 0.85)
- Building Higher-Order Abstractions from the Components of Recommender Systems (AAAI 2024 · similarity 0.85)
- Tree of Preferences for Diversified Recommendation (NeurIPS 2025 · similarity 0.85)
- Recommender Ecosystems: A Mechanism Design Perspective on Holistic Modeling and Optimization (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.