Research map / Label-efficient and robust learning / Recognition and long-tail
Dual-View Whitening on Pre-trained Text Embeddings for Sequential Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DWSRec, a text-only sequential recommender that applies fully and relaxed (group) ZCA whitening to pre-trained text embeddings as dual views, fed into a decoupled-attention dual-view transformer.
Problem
Pre-trained text embeddings used in sequential recommendation are anisotropic (degenerate), and full whitening can distort semantic similarity among items with similar text, limiting recommendation performance without ID embeddings.
Approach
Apply ZCA whitening at two strengths (full G=1 and relaxed group whitening G>1) to text embeddings, encode both views with a decoupled-attention dual-view transformer that swaps key-query pairs across views, and fuse views with learnable attention.
Key results
- DWSRec achieves R@20 of 0.1710 on Arts vs 0.1625 for SASRecW and 0.1611 for UniSRecT+ID under warm-start.
- On Toys, DWSRec reaches R@50 0.1931 vs 0.1798 for SASRecW; on Tools R@50 0.1254 vs 0.1196.
- Full whitening (SASRecW) improves over raw text features SASRecT, e.g. Arts R@50 0.2348 vs 0.2129.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add full whitening to pre-trained text embeddings | SASRec text embeddings · Arts, Toys, Tools | R@50: 0.2129 -> 0.2348; 0.1542 -> 0.1798; 0.1055 -> 0.1196 | Helped |
| Use both full and relaxed whitening views with dual-view encoders | Sequential recommendation · Arts, Toys, Tools | R@20: 0.1625 -> 0.1710; 0.1201 -> 0.1307; 0.0861 -> 0.0918 | Helped |
| Add s1 (full-whitening sequence embedding) (ablation) | DWSRec · Arts, Toys, Tools | R@20: 0.1650 -> 0.1710; 0.1217 -> 0.1307; 0.0866 -> 0.0918 | Helped |
| Add s2 (relaxed-whitening sequence embedding) (ablation) | DWSRec · Arts, Toys, Tools | R@20: 0.1652 -> 0.1710; 0.1241 -> 0.1307; 0.0869 -> 0.0918 | Helped |
| Use decoupled attention instead of a conventional transformer (ablation) | DWSRec · Arts, Toys, Tools | R@20: 0.1647 -> 0.1710; 0.1222 -> 0.1307; 0.0884 -> 0.0918 | Helped |
| Use attention fusion instead of element-wise summation (ablation) | DWSRec · Arts, Toys, Tools | R@20: 0.1702 -> 0.1710; 0.1253 -> 0.1307; 0.0908 -> 0.0918 | Helped |
| Use relaxed whitening (G>1) rather than full whitening (G=1) | SASRecW · Arts, Toys | R@50 | Hurt |
| Use relaxed whitening for cold-start recommendations | SASRecW · Arts, Toys, Tools | R@20 | Helped |
| Use dual-view whitening for cold-start recommendations | DWSRec · Arts, Toys, Tools | R@20 | Helped |
| Use PCA whitening instead of CD or ZCA whitening | DWSRec · Arts, Toys, Tools | N@20 | Hurt |
| Use parametric whitening instead of non-parametric whitening | DWSRec · Arts, Toys, Tools | N@20 | Hurt |
| Increase relaxed-whitening group count | DWSRec · Arts, Toys, Tools | N@20 | Mixed |
Surprising findings
- Relaxed whitening alone degrades performance relative to full whitening in warm-start settings, while raw unwhitened text features perform worst across datasets.
- Text-only SASRecW matches or exceeds variants that add ID embeddings (SASRecT+ID, UniSRecT+ID) without learnable ID parameters.
Limitations
- Relaxed whitening group number G must be tuned and the optimal value differs by dataset.
- Evaluated only on three Amazon review datasets with a single text encoder (BERT).
Experimental setup
Datasets: Amazon Arts, Crafts and Sewing Amazon Toys and Games Amazon Tools and Instruments
Benchmarks: Amazon Arts Amazon Toys Amazon Tools
Metrics: Recall@K NDCG@K
Baselines: GRCN BM3 SASRecID HGN CL4SRec SASRecT SASRecT+ID FDSA S3-Rec DIF-SR UniSRecT UniSRecT+ID
Builds on: SASRec UniSRec DIF-SR ZCA whitening group whitening
Closest papers by meaning
- Augmenting Sequential Recommendation with Balanced Relevance and Diversity (AAAI 2025 · similarity 0.87)
- LLMEmb: Large Language Model Can Be a Good Embedding Generator for Sequential Recommendation (AAAI 2025 · similarity 0.86)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.86)
- SRA-CL: Semantic Retrieval Augmented Contrastive Learning for Sequential Recommendation (NeurIPS 2025 · similarity 0.86)
- Future Sight and Tough Fights: Revolutionizing Sequential Recommendation with FENRec (AAAI 2025 · similarity 0.86)
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.