Research map / Architectures and efficiency / RNNs and normalization
Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Generative Recommenders (GRs) that reformulate ranking and retrieval as sequential transduction over interleaved user actions and items, and HSTU, a pointwise-attention encoder, scaling to 1.5T parameters with a power-law scaling law and a 12.4% online A/B gain.
Problem
Industrial DLRM recommenders depend on heterogeneous hand-engineered features and fail to scale with compute, while sequential transduction models are not practical at billion-user, billion-vocabulary, streaming scale.
Approach
Treat user actions as a new modality and cast ranking and retrieval as generative sequential transduction over interleaved item-action sequences, with a HSTU encoder that replaces softmax attention with pointwise aggregated attention and gating, plus stochastic length sparsification and M-FALCON for cheap candidate scoring.
Key results
- HSTU-large improves NDCG@10 by 65.8% over SASRec on Amazon Books and 18.1% on ML-1M in public multi-pass settings.
- HSTU is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192-length sequences in training, per the abstract.
- Deployed GR ranking model yields +12.4% online E-task gain in A/B tests, with 1.5 trillion parameters and power-law scaling across three orders of magnitude of compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SASRec with HSTU-large | Sequential recommendation · ML-1M | NDCG@200: .2498 -> .2771 | Helped |
| Replace SASRec with HSTU-large | Sequential recommendation · ML-20M | NDCG@200: .2521 -> .2971 | Helped |
| Replace SASRec with HSTU-large | Sequential recommendation · Books | NDCG@200: .0350 -> .0508 | Helped |
| Use HSTU instead of Transformer++ | Industrial retrieval · Industrial-scale one-pass streaming | log perplexity: 4.015 -> 3.978 | Helped |
| Use HSTU instead of Transformer++ | Industrial ranking · Industrial-scale one-pass streaming | Normalized Entropy (NE), E-Task: .4945 -> .4937 | Helped |
| Use HSTU instead of Transformer++ | Industrial ranking · Industrial-scale one-pass streaming | Normalized Entropy (NE), C-Task: .7822 -> .7805 | Helped |
| Use GR instead of DLRM | Retrieval · Industrial offline retrieval | HR@100: 29.0% -> 36.9% | Helped |
| Use GR instead of DLRM | Ranking · Industrial offline ranking E-Task | Normalized Entropy (NE): .4982 -> .4845 | Helped |
| Use GR instead of DLRM | Ranking · Industrial offline ranking C-Task | Normalized Entropy (NE): .7842 -> .7645 | Helped |
| Replace HSTU pointwise attention with softmax (ablation) | HSTU · Synthetic streaming | HR@10: .0893 -> .0617 | Hurt |
| Add relative attention bias rab^{p,t} (ablation) | HSTU · Industrial retrieval | log perplexity: 4.021 -> 3.978 | Helped |
| Add relative attention bias rab^{p,t} (ablation) | HSTU · Industrial ranking E-Task | Normalized Entropy (NE): .4980 -> .4937 | Helped |
Surprising findings
- Standard Transformers suffered frequent loss explosions in industrial ranking, requiring ~10x lower learning rates for softmax-based variants.
- DLRMs given the same handful of raw features as GRs degrade significantly, while GRs with 285x higher complexity achieve 1.5-3x higher QPS.
Limitations
- Industrial experiments use proprietary data and infrastructure, limiting reproducibility
- Full-shuffle multi-epoch public benchmark evaluation differs from streaming industrial settings
- Stochastic length introduces sparsity hyperparameter alpha that must be tuned per sequence length
Experimental setup
Datasets: MovieLens-1M MovieLens-20M Amazon Books industrial streaming logs synthetic Dirichlet process data
Benchmarks: MovieLens-1M MovieLens-20M Amazon Books
Metrics: HR@10 HR@50 HR@200 NDCG@10 NDCG@200 normalized entropy log perplexity online A/B lift
Baselines: SASRec GRU4Rec BERT4Rec Transformers Transformer++ DLRM DHEN DCN
Builds on: Transformer SASRec FlashAttention-2 DLRM SwiGLU
Closest papers by meaning
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.88)
- Multimodal Quantitative Language for Generative Recommendation (ICLR 2025 · similarity 0.88)
- Echoes in Filter Bubble: Diagnosing and Curing Popularity Bias in Generative Recommender Systems (NeurIPS 2026 · similarity 0.87)
- Wukong: Towards a Scaling Law for Large-Scale Recommendation (ICML 2024 · similarity 0.87)
- Preference Diffusion for Recommendation (ICLR 2025 · similarity 0.87)
- ConvRec-R1: Training LLM-based Conversational Recommender Systems with Reinforcement Learning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.