Research map / Architectures and efficiency / Attention and transformers
Iterative Sparse Attention for Long-sequence Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ISA, a sparse-attention plus personalized-PageRank iterative-attention model for long-sequence CTR/sequential recommendation that cuts attention cost while expanding receptive fields to capture interest drift.
Problem
Long user behavior sequences make self-attention costly for recommendation, and interest drift between distant historical items is hard to capture.
Approach
Combines global, window, and random sparse attention (causal-masked, with individual-item selection) with a Personalized PageRank power iteration that diffuses sparse attention over multiple hops, so distant items connect and the output approximates full attention.
Key results
- On Taobao, ISA reaches AUC 0.8849 and GAUC 0.8848, outperforming the best baseline by 1.96% GAUC (per paper's claim against other baselines).
- On Short Video, ISA reaches GAUC 0.8147 and AUC 0.8079, improving over Bigbird4Rec (GAUC 0.8098) and SASRec-style baselines.
- Ablation shows removing the Sparse Attention Layer hurts most, and removing the Iterative Attention Layer also degrades GAUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ISA instead of Bigbird4Rec | Long-sequence recommendation · Taobao | AUC: 0.8655 -> 0.8849 | Helped |
| Use ISA instead of ETA | Long-sequence recommendation · Taobao | MRR: 0.4064 -> 0.4314 | Helped |
| Use ISA instead of ETA | Long-sequence recommendation · Taobao | NDCG: 0.3492 -> 0.3751 | Helped |
| Use ISA instead of Bigbird4Rec | Long-sequence recommendation · Taobao | GAUC: 0.8653 -> 0.8848 | Helped |
| Use ISA instead of Long4Rec | Long-sequence recommendation · Short Video | AUC: 0.8014 -> 0.8079 | Helped |
| Use ISA instead of SASRec | Long-sequence recommendation · Short Video | MRR: 0.9063 -> 0.9088 | Helped |
| Use ISA instead of SASRec | Long-sequence recommendation · Short Video | NDCG: 0.9308 -> 0.9327 | Helped |
| Use ISA instead of Long4Rec | Long-sequence recommendation · Short Video | GAUC: 0.8102 -> 0.8147 | Helped |
| Add Sparse Attention Layer (SAL) (ablation) | ISA · Taobao and Short Video | GAUC | Helped |
| Add Iterative Attention Layer (IAL) (ablation) | ISA with sparse attention · Taobao and Short Video | GAUC | Helped |
Surprising findings
- Sparse attention baselines (Long4Rec, Bigbird4Rec) outperform full-attention SASRec on Taobao, which the authors attribute to overfitting from redundant full attention on sparse data.
Limitations
- Experiments limited to two datasets; the authors note future verification is needed when rich user/item attributes are available.
- Sparse attention and multi-hop diffusion may miss some relations; approximation to full attention is argued theoretically rather than fully proven empirically.
Experimental setup
Datasets: Taobao Short Video
Benchmarks: Taobao Short Video
Metrics: AUC GAUC MRR NDCG@10
Baselines: DIN SIM ETA UBR4CTR SAM SDIM SASRec Long4Rec Bigbird4Rec
Builds on: Longformer BigBird SAM SASRec DIN
Closest papers by meaning
- Long-Sequence Recommendation Models Need Decoupled Embeddings (ICLR 2025 · similarity 0.88)
- An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention (AAAI 2024 · similarity 0.88)
- TV-Rec: Time-Variant Convolutional Filter for Sequential Recommendation (NeurIPS 2025 · similarity 0.87)
- Sparse Enhanced Network: An Adversarial Generation Method for Robust Augmentation in Sequential Recommendation (AAAI 2024 · similarity 0.87)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.86)
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.