Research map / Architectures and efficiency / Attention and transformers

An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

AAAI 2024 · Architectures and efficiency › Attention and transformers · approach: Sparse and structured attention · 124 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BSARec, which mixes self-attention with a Fourier-domain inductive-bias filter that keeps low frequencies and learnably rescales high frequencies, to counter the low-pass oversmoothing of Transformer sequential recommenders.

Problem

Transformer-based sequential recommenders lack inductive bias and act as low-pass filters, causing oversmoothing of hidden representations and loss of short-term, high-frequency user preference signals.

Approach

Combine learned self-attention with a fixed DFT-based attentive inductive bias in which low-frequency components are kept and high-frequency components are scaled by a trainable parameter beta, so high-pass information is retained alongside self-attention.

Key results

• BSARec achieves best HR@10 on all six datasets versus baselines, e.g., +27.49% HR@10 over the best baseline on LastFM and +4.10% HR@5 on Beauty.
• BSARec outperforms contrastive-learning models DuoRec and FEARec without using contrastive learning across all datasets and metrics (full-ranking evaluation).
• Ablation on Beauty: using only the inductive-bias term gives HR@20 0.1338 vs 0.1265 for self-attention only; the full model reaches 0.1373.

Measured effects reported in the paper

Surprising findings

• Using only the fixed inductive-bias filter outperforms using only self-attention on Beauty and Toys.
• The simple architecture without contrastive learning beats the more complex contrastive-learning baselines DuoRec and FEARec.

Limitations

• Hyperparameters such as alpha and c must be tuned per dataset, and the best values vary widely.
• The low-pass analysis relies on a theorem whose proof is informal (Perron-Frobenius argument with asymptotic hand-waving).

Experimental setup

Datasets: Amazon Beauty Amazon Sports Amazon Toys Yelp LastFM ML-1M

Benchmarks: Amazon Beauty Amazon Sports Amazon Toys Yelp LastFM ML-1M

Metrics: HR@5 HR@10 HR@20 NDCG@5 NDCG@10 NDCG@20 MRR

Baselines: Caser GRU4Rec SASRec BERT4Rec FMLPRec DuoRec FEARec

Builds on: SASRec FMLPRec FEARec DuoRec

Closest papers by meaning

• TV-Rec: Time-Variant Convolutional Filter for Sequential Recommendation (NeurIPS 2025 · similarity 0.89)
• Iterative Sparse Attention for Long-sequence Recommendation (AAAI 2025 · similarity 0.88)
• SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.87)
• LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.86)
• Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.85)
• LLM4RSR: Large Language Models as Data Correctors for Robust Sequential Recommendation (AAAI 2025 · similarity 0.85)