Research map / Label-efficient and robust learning / Recognition and long-tail
On the Embedding Collapse when Scaling up Recommendation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies embedding collapse in recommendation models as a barrier to scaling, explains it via feature interaction, and proposes multi-embedding (multiple embedding sets with embedding-set-specific interaction modules) that improves scalability and mitigates collapse.
Problem
Recommendation models fail to improve, or even degrade, when their embedding size is scaled up, because learned embedding matrices collapse into low-dimensional subspaces.
Approach
Feature interaction causes embedding collapse, yet removing interaction causes overfitting; so instead of suppressing interaction, scale up several independent embedding sets, each with its own interaction module, and average their outputs, which yields more diverse, less-collapsed embeddings.
Key results
- Multi-embedding with 10x size improves DCNv2 test AUC by 0.00099 on Criteo and 0.00223 on Avazu, while single-embedding gives little or negative scaling gains.
- Multi-embedding gives consistent scaling across 6 mainstream models; the largest ME models average 0.00110 AUC improvement over baselines.
- Deployment in Tencent's advertising platform (WeChat Moments) yielded a 3.9% GMV lift in online A/B testing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace single-embedding with multi-embedding and scale model size to 10x | Recommendation models on Criteo · Criteo test | AUC: 0.00110 improvement on the test AUC | Helped |
| Replace single-embedding with multi-embedding and scale model size to 10x | Recommendation models on Avazu · Avazu test | AUC: 0.00223 on Avazu | Helped |
| Increase embedding size to 10x with single-embedding | DNN, xDeepFM, and FinalMLP · Criteo test | AUC: 0.00136 on Criteo | Hurt |
| Increase embedding size to 10x with single-embedding | DNN, xDeepFM, and FinalMLP · Avazu test | AUC: 0.00118 on Avazu | Hurt |
| Regularize DCNv2 interaction projections to preserve singular values | DCNv2 · Criteo | AUC | No clear effect |
| Suppress explicit feature interaction using DNN | DNN · Recommendation model scaling | AUC | Hurt |
| Add embedding-set-specific interaction modules rather than sharing interaction (ablation) | Multi-embedding DCNv2 · Recommendation model scaling | AUC | Helped |
| Restrict divergence of interaction weights across embedding sets (ablation) | Multi-embedding DCNv2 · Recommendation model scaling | AUC | Hurt |
| Increase the number of cross layers (ablation) | DCNv2 · Criteo test | AUC: 0.81339 -> 0.81325; 0.81339 -> 0.81338; 0.81339 -> 0.81344 | No clear effect |
| Increase MLP depth (ablation) | DCNv2 · Criteo test | AUC: 0.81339 -> 0.81337; 0.81339 -> 0.81345; 0.81339 -> 0.81342 | No clear effect |
| Use multi-embedding rather than single-embedding | AutoInt · Criteo test | AUC: 6e-4 AUC improvement | Helped |
| Add a nonlinear projection to the interaction module (ablation) | Multi-embedding with linear interaction · Recommendation models | AUC | Helped |
Surprising findings
- Regularizing projections to preserve information abundance or suppressing feature interaction (e.g., DNN) avoids collapse but causes overfitting and no AUC scaling.
- Scaling only the interaction or prediction modules (more cross or MLP layers) does not improve performance; embeddings are the key bottleneck.
Limitations
- Analysis is restricted to CTR prediction models on Criteo and Avazu with a fixed set of architectures.
- Multi-embedding requires non-linear interaction modules; linear interaction models are equivalent to single-embedding otherwise.
- Theoretical analysis is limited to an FM-style interaction and the first cross layer, as a simplification.
Experimental setup
Datasets: Criteo Avazu
Benchmarks: Criteo Avazu
Metrics: test AUC information abundance principal-angle-based embedding diversity
Baselines: DNN IPNN NFwFM xDeepFM DCNv2 FinalMLP AutoInt
Builds on: DCNv2 FwFM xDeepFM FinalMLP FFM
Closest papers by meaning
- Long-Sequence Recommendation Models Need Decoupled Embeddings (ICLR 2025 · similarity 0.89)
- FACE: A General Framework for Mapping Collaborative Filtering Embeddings into LLM Tokens (NeurIPS 2025 · similarity 0.87)
- LIME: Link-based User-item Interaction Modeling with Decoupled XOR Attention for Efficient Test Time Scaling (NeurIPS 2026 · similarity 0.86)
- Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution (NeurIPS 2024 · similarity 0.86)
- Wukong: Towards a Scaling Law for Large-Scale Recommendation (ICML 2024 · similarity 0.86)
- Dual-View Whitening on Pre-trained Text Embeddings for Sequential Recommendation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.