Research map / Label-efficient and robust learning / Recognition and long-tail
Both Supply and Precision: Sample Debias and Ranking Consistency Joint Learning for Large Scale Pre-Ranking System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDCL, an end-to-end multi-task framework for the pre-ranking stage of Weibo's feed recommender that combines multi-task distillation from a jointly trained ranking model with adaptive weighting of supplied negative samples to address sample selection bias and ranking consistency.
Problem
Pre-ranking in cascade recommenders suffers from sample selection bias (training on exposed items only while inferring over a much larger item space) and ranking consistency problems (pre-ranking and ranking orders diverge after multi-task score fusion).
Approach
Jointly train pre-ranking and ranking nets with shared feature embeddings and per-task logit distillation from the ranking net, while supplying negatives from upstream stages and weighting them adaptively by a similarity term and the current model's confidence, all in one multi-task objective.
Key results
- SDCL improves Precision@100 from 0.29458 (SNR) to 0.35023 and reduces Reverse Rate from 0.4393 to 0.3631 in offline evaluation.
- In a one-week online A/B test against SNR, SDCL gives +2.57% Average Watch Time, +1.38% Average Number of Valid Reads, and +1.41% Average Number of Requests.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-task distillation (ablation) | SNR pre-ranking model · Pre-ranking evaluation | Reverse Rate: 0.4393 -> 0.4254 | Helped |
| Add multi-task distillation (ablation) | SNR pre-ranking model · Pre-ranking evaluation | Recall@100: 0.29458 -> 0.33074 | Helped |
| Add multi-task distillation (ablation) | SNR pre-ranking model · Pre-ranking evaluation | Precision@100: 0.01222 -> 0.01200 | Hurt |
| Add negative samples with fixed weights (ablation) | SNR w/ MTD · Pre-ranking evaluation | Recall@100: 0.33074 -> 0.33963 | Helped |
| Add negative samples with fixed weights (ablation) | SNR w/ MTD · Pre-ranking evaluation | Precision@100: 0.01200 -> 0.01543 | Helped |
| Add negative samples with fixed weights (ablation) | SNR w/ MTD · Pre-ranking evaluation | Reverse Rate: 0.4254 -> 0.3634 | Helped |
| Replace fixed negative-sample weights with adaptive weights (ablation) | SNR w/ MTD & fixed weights · Pre-ranking evaluation | Recall@100: 0.33963 -> 0.35023 | Helped |
| Replace fixed negative-sample weights with adaptive weights (ablation) | SNR w/ MTD & fixed weights · Pre-ranking evaluation | Precision@100: 0.01543 -> 0.01719 | Helped |
| Replace fixed negative-sample weights with adaptive weights (ablation) | SNR w/ MTD & fixed weights · Pre-ranking evaluation | Reverse Rate: 0.3634 -> 0.3631 | Helped |
| Use 18 rather than 8 negative samples (ablation) | SNR · Pre-ranking evaluation, CTR | AUC: 0.838 -> 0.852(+1.7%) | Helped |
| Use 18 rather than 8 negative samples (ablation) | SNR · Pre-ranking evaluation, TIME | AUC: 0.837 -> 0.854(+2.0%) | Helped |
| Use 18 rather than 8 negative samples (ablation) | SNR · Pre-ranking evaluation, ITR | AUC: 0.959 -> 0.946(-1.4%) | Hurt |
Surprising findings
- Adding multi-task distillation alone lowered Recall@100 slightly (0.04393 to 0.4254 reverse rate change aside, Recall@100 went 0.01222 to 0.01200), while improving Precision@100.
- Negative samples from ranking stage (the hardest) yielded lowest AUC, indicating higher learning difficulty.
Limitations
- Evaluated only on a single proprietary industrial dataset with no public pre-ranking benchmark.
- Matching stage is not included in the joint framework.
Experimental setup
Datasets: Weibo homepage recommendation production traffic logs
Benchmarks: Weibo internal traffic logs (offline) Weibo online A/B test
Metrics: Precision@100 Recall@100 Reverse Rate AUC Average Watch Time Average Number of Requests Average Number of Valid Reads
Baselines: SNR
Builds on: SNR PEPNet FSCD Knowledge distillation (logit-based)
Closest papers by meaning
- From Pairwise to Ranking: Climbing the Ladder to Ideal Collaborative Filtering with Pseudo-Ranking (AAAI 2025 · similarity 0.85)
- Accelerated Spectral Ranking (ICML 2018 · similarity 0.84)
- Learning Cascade Ranking as One Network (ICML 2025 · similarity 0.84)
- Relaxing the Accurate Imputation Assumption in Doubly Robust Learning for Debiased Collaborative Filtering (ICML 2024 · similarity 0.83)
- Fine-Grained Dynamic Framework for Bias-Variance Joint Optimization on Data Missing Not at Random (NeurIPS 2024 · similarity 0.82)
- Learning-Efficient Yet Generalizable Collaborative Filtering for Item Recommendation (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.