Research map / Architectures and efficiency / RNNs and normalization
Wukong: Towards a Scaling Law for Large-Scale Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Wukong, a recommendation architecture of stacked factorization machine blocks plus linear compression blocks with a scaling strategy, showing outperformance on six public CTR datasets and a scaling law up to 100+ GFLOP/example on a 146B-sample internal dataset.
Problem
Recommendation models do not show scaling laws like LLMs; prior dense interaction architectures saturate or become unstable when upscaled, and sparse embedding scaling is costly and misaligned with accelerator compute.
Approach
Stack factorization machine blocks (FMB) with linear compression blocks (LCB) and residual connections so that layer i captures interactions of order 1 to 2^i (binary-exponentiation-like growth), with an optimized low-rank FM and a pyramid layer shape, enabling dense scaling.
Key results
- Wukong achieves the best AUC on all six public datasets, e.g., 0.7292 AUC on MicroVideo vs 0.7255 for the best baseline MaskNet.
- On the internal 146B-sample dataset, Wukong keeps a scaling trend across two orders of magnitude of complexity, reaching 0.76% relative LogLoss gain at 108 GFLOP/example and 644B parameters.
- Wukong gives about 0.1% LogLoss improvement per quadrupling of complexity, and DCNv2 would need about 40x more complexity to match Wukong's quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Wukong instead of the strongest AUC baseline | Recommendation models · Frappe | AUC: 0.9868 -> 0.9868 | No clear effect |
| Use Wukong instead of the strongest AUC baseline | Recommendation models · MicroVideo | AUC: 0.7255 -> 0.7292 | Helped |
| Use Wukong instead of the strongest AUC baseline | Recommendation models · MovieLens Latest | AUC: 0.9723 -> 0.9723 | No clear effect |
| Use Wukong instead of the strongest AUC baseline | Recommendation models · KuaiVideo | AUC: 0.7376 -> 0.7414 | Helped |
| Use Wukong instead of the strongest AUC baseline | Recommendation models · TaobaoAds | AUC: 0.6457 -> 0.6488 | Helped |
| Use Wukong instead of the strongest AUC baseline | Recommendation models · Criteo Terabyte | AUC: 0.8100 -> 0.8106 | Helped |
| Add a Factorization Machine Block (FMB) (ablation) | Wukong Interaction Stack · Internal dataset, Task1 | Relative LogLoss | Helped |
| Add a Linear Compression Block (LCB) (ablation) | Wukong Interaction Stack · Internal dataset, Task1 | Relative LogLoss | Helped |
| Add residual connections (ablation) | Wukong Interaction Stack · Internal dataset, Task1 | Relative LogLoss | Helped |
| Add both LCB and residual connections (ablation) | Wukong Interaction Stack · Internal dataset, Task1 | Relative LogLoss | Helped |
| Increase the number of Interaction Stack layers (ablation) | Wukong · Internal dataset, Task1 | Relative LogLoss: -0.35 -> -0.45 -> -0.53 | Helped |
| Increase the FMB MLP size (ablation) | Wukong · Internal dataset, Task1 | Relative LogLoss | Helped |
Surprising findings
- Scaled-up AutoInt+, DCNv2 and MaskNet suffered training instability or out-of-memory errors, and AFN+ scaling did not improve quality.
- Applying Wukong's components (bit-wise MLP, pyramid shape) to AutoInt+ gave a 0.57% LogLoss gain at about 5 GFLOP/example, versus 50 GFLOP/example for scaled-up AutoInt+.
Limitations
- Upper limit of scalability not reached due to compute requirements.
- Theoretical understanding of the architecture remains limited.
- Generalizability beyond recommendation not explored; the internal dataset is proprietary and large-model training was not converged.
Experimental setup
Datasets: Frappe MicroVideo MovieLens Latest KuaiVideo TaobaoAds Criteo Terabyte Internal dataset (146B entries, 720 features)
Benchmarks: Frappe MicroVideo MovieLens Latest KuaiVideo TaobaoAds Criteo Terabyte Internal dataset
Metrics: AUC LogLoss Relative LogLoss GFLOP/example parameter count
Baselines: AFN+ AutoInt+ DLRM DCNv2 FinalMLP MaskNet xDeepFM
Builds on: DLRM DCNv2 AutoInt+ FinalMLP MaskNet
Closest papers by meaning
- Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (ICML 2024 · similarity 0.87)
- LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling (NeurIPS 2026 · similarity 0.86)
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.86)
- On the Embedding Collapse when Scaling up Recommendation Models (ICML 2024 · similarity 0.86)
- ContextGNN: Beyond Two-Tower Recommendation Systems (ICLR 2025 · similarity 0.85)
- Tree of Preferences for Diversified Recommendation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.