Research map / Label-efficient and robust learning / Recognition and long-tail
RecFlow: An Industrial Full Flow Recommendation Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RecFlow, an industrial dataset that records unexposed items filtered at each of six recommendation funnel stages alongside exposure samples, and shows stage samples help retrieval, coarse ranking and ranking experiments.
Problem
Existing recommendation benchmarks only include exposed items, so they miss the unexposed serving space and the interplay between multi-stage pipeline components, creating a train-serving gap for industrial RS algorithms.
Approach
A full-flow dataset that logs stage-wise filtered candidates (pre-rank, coarse rank, rank, re-rank negatives/positives) per request, enabling hard-negative mining, FS-LTR priority ranking loss across stages, and stage-aware auxiliary losses and evaluation.
Key results
- In retrieval, using rerank_pos stage samples as hard negatives raised R@100 from 0.461% to 0.687%, and FS-LTR with stage samples reached R@100 0.803%, a relative gain of about 74%.
- For coarse ranking, adding all stage samples as 10 extra negatives raised Recall@100 from 0.271 to 0.455 while AUC dropped from 0.718 to 0.563.
- The coarse-ranking auxiliary ranking loss raised AUC from 0.718 to 0.721 and reduced LogLoss from 0.592 to 0.588.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a random negative with a single stage sample as a hard negative (ablation) | SASRec retrieval · Retrieval | Recall@100: Baseline 0.461±0.085; stage variants 0.462±0.126 to 0.687±0.087 | Mixed |
| Use rerank_pos instead of random negatives as hard negatives (ablation) | SASRec retrieval · Retrieval | Recall@100: 0.461±0.085 -> 0.687±0.087 | Helped |
| Increase hard negatives to 10 for several stage types (ablation) | SASRec retrieval · Retrieval | Recall@100: Rank_neg: 0.492 -> 0.323; Rerank_neg: 0.606 -> 0.336; Rerank_pos: 0.428 -> 0.219 | Hurt |
| Add stage-priority ranking loss (FS-LTR) (ablation) | SASRec retrieval · Retrieval | Recall@100: 0.461±0.085 -> 0.803±0.095 | Helped |
| Use cascade samples from all stages rather than only coarse_neg and pre-rank_neg (ablation) | FS-LTR retrieval · Retrieval | Recall@100: CN-PN 0.803; All 0.663 | Hurt |
| Add extra stage samples as negative training examples (ablation) | DSSM coarse ranking · Coarse ranking candidate set | Recall@100: 0.271±0.027 -> 0.455±0.004 | Helped |
| Add extra stage negatives to coarse ranking (ablation) | DSSM coarse ranking · Coarse ranking exposure test set | AUC: 0.718±0.001 -> 0.563±0.004 | Hurt |
| Add stage-priority ranking loss (FS-LTR) (ablation) | DSSM coarse ranking · Coarse ranking candidate set | Recall@100: Baseline 0.271±0.027; FS-LTR 0.475±0.002; PositiveRank 0.457±0.001 | Helped |
| Add an auxiliary ranking loss using stage samples (ablation) | DSSM coarse ranking · Coarse ranking exposure test set | AUC: 0.718 -> 0.721 | Helped |
| Model competitive context from rank_pos samples in the behavior sequence (ablation) | DSSM coarse ranking · Coarse ranking exposure test set | AUC: 0.718 -> 0.722 | Helped |
| Add extra stage samples as negative training examples (ablation) | DIN ranking · Ranking candidate set | Recall@50: 0.169±0.005 -> 0.278±0.027 | Helped |
| Add extra stage negatives to ranking (ablation) | DIN ranking · Ranking exposure test set | AUC: 0.727±0.001 -> 0.602±0.005 | Hurt |
Surprising findings
- Adding extra stage negatives improves Recall/NDCG while substantially degrading the classical AUC metric.
- Hard negatives from stages closer to the positive feedback degrade retrieval performance when used in larger numbers.
Limitations
- Data comes from only one recommendation scenario, so multi/cross-domain recommendation is not supported.
- No multimodal features such as text or images.
- Large scale (1.9B instances) demands substantial hardware resources.
Experimental setup
Datasets: RecFlow
Benchmarks: RecFlow
Metrics: AUC LogLoss Recall@K NDCG@K
Baselines: SASRec with random negatives DSSM baseline DIN baseline PositiveRank
Builds on: FS-LTR SASRec DSSM DIN BPR
Closest papers by meaning
- ORBIT - Open Recommendation Benchmark for Reproducible Research with Hidden Tests (NeurIPS 2025 · similarity 0.88)
- Identify Then Recommend: Towards Unsupervised Group Recommendation (NeurIPS 2024 · similarity 0.87)
- LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application (AAAI 2025 · similarity 0.86)
- Building Higher-Order Abstractions from the Components of Recommender Systems (AAAI 2024 · similarity 0.86)
- AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems (NeurIPS 2025 · similarity 0.86)
- AT4CTR: Auxiliary Match Tasks for Enhancing Click-Through Rate Prediction (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.