Research map / Architectures and efficiency / Attention and transformers
Long-Sequence Recommendation Models Need Decoupled Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DARE, which uses two separate embedding tables (one for attention, one for representation) in long-sequence CTR models to fix gradient domination and conflict between the two modules, improving AUC and enabling faster search.
Problem
Long-sequence recommendation models share one embedding table for both the attention (behavior retrieval) and representation modules, causing gradient domination and conflict that degrade both attention accuracy and representation discriminability; common fixes like linear projections fail due to lo
Approach
Decouple attention and representation at the embedding level by using two independent embedding tables, with target-aware representation (element-wise product) for the representation branch; this also allows a smaller attention embedding dimension to accelerate the search stage.
Key results
- DARE achieves AUC 0.92568 on Taobao and 0.96800 on Tmall at embedding dim 16, versus 0.91688 and 0.95812 for TWIN, a gain of about 9 and 6 per thousand.
- Retrieval NDCG of DARE is 0.8124 versus 0.5545 for TWIN (46.5% relative increase).
- In a 5-day online A/B test on Tencent's advertising platform, DARE gave a 1.47% GMV lift and 0.57% cost reduction over TWIN; reducing the attention dimension speeds search by 50% with little AUC loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decouple attention and representation into separate embedding tables | TWIN · Taobao and Tmall, embedding dimensions 16, 64, and 128 | AUC: AUC improvements of 1\textperthousand~and 9\textperthousand~ | Helped |
| Decouple attention and representation embeddings | TWIN · Behavior retrieval on test data | NDCG: 0.8124 vs. 0.5545 | Helped |
| Decouple attention and representation embeddings | TWIN · Behavior retrieval on test data | NDCG: 0.8124 vs. 0.6382 | Helped |
| Reduce the attention embedding dimension (ablation) | DARE · Long-sequence inference | search speed: 50% | Helped |
| Reduce the attention embedding dimension (ablation) | DARE · Long-sequence inference | AUC | No clear effect |
| Reduce the attention embedding dimension further (ablation) | DARE · Long-sequence inference | search speed: 75% | Helped |
| Remove target-aware representation (ablation) | TWIN · Taobao and Tmall | AUC: 0.91688 -> 0.90732; 0.95812 -> 0.96170 | Mixed |
| Replace shared history/target embeddings with four independent embeddings (ablation) | TWIN · Taobao and Tmall | AUC: 0.91688 -> 0.90414; 0.95812 -> 0.96124 | Hurt |
| Add linear projections for attention and representation (ablation) | TWIN · Taobao and Tmall | AUC: 0.91688 -> 0.89642; 0.95812 -> 0.96152 | Mixed |
| Apply linear projection to category features (ablation) | TWIN · Taobao and Tmall | AUC: 0.91688 -> 0.91738; 0.95812 -> 0.96658 | Helped |
| Replace linear projection with MLP projection (ablation) | TWIN · Taobao and Tmall | AUC: 0.91688 -> 0.86960; 0.95812 -> 0.95678 | Hurt |
| Use gradient normalization to avoid representation-gradient domination (ablation) | TWIN · Taobao and Tmall | AUC: 0.91688 -> 0.91976; 0.95812 -> 0.93887 | Mixed |
Surprising findings
- Linear projections, which help in NLP, cause an AUC drop of nearly 2% on Taobao when applied to TWIN.
- Gradients from the representation module are about five times larger than attention gradients on shared embeddings, and in nearly two-thirds of cases the two gradients conflict.
Limitations
- Advantage over TWIN shrinks with shorter sequences, where representation dominance matters less.
- Short-sequence gains on Amazon are marginal (0.86309 vs 0.86291 AUC for TIN).
- Some baselines (e.g., TWIN-V2) were adapted without their domain-specific features, so comparisons may be sensitive to implementation.
Experimental setup
Datasets: Taobao Tmall Amazon
Benchmarks: Taobao Tmall Amazon
Metrics: AUC GAUC Logloss NDCG mutual information GMV lift
Baselines: ETA SDIM DIN TWIN TWIN (hard) TWIN (w/ proj.) TWIN (w/o TR) TWIN-V2 TWIN-4E
Builds on: TWIN SIM DIN TIN Transformer self-attention
Closest papers by meaning
- On the Embedding Collapse when Scaling up Recommendation Models (ICML 2024 · similarity 0.89)
- Iterative Sparse Attention for Long-sequence Recommendation (AAAI 2025 · similarity 0.88)
- UxSID: Semantic-Aware User Interests Modeling for Ultra-Long Sequence (NeurIPS 2026 · similarity 0.88)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.87)
- Enhancing Long-and Short-Term Representations for Next POI Recommendations via Frequency and Hierarchical Contrastive Learning (AAAI 2025 · similarity 0.87)
- FACE: A General Framework for Mapping Collaborative Filtering Embeddings into LLM Tokens (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.