Research map / Vision-language and multimodal / Cross-modal learning
MTSTRec: Multimodal Time-Aligned Shared Token Recommender
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MTSTRec, a transformer for multimodal sequential recommendation that fuses item ID, Gram-matrix style, LLM text, LLM-prompt text and price via one time-aligned shared token per item per layer.
Problem
Sequential recommendation in e-commerce often ignores temporal alignment across modalities (text, image, price, ID) in browsing sequences, and existing early/late fusion methods fail to capture cross-modal interactions per time step.
Approach
A transformer with separate per-modality encoders plus a time-aligned shared token per sequence position, averaged across modalities at each layer, so that each item's modality tokens exchange information at the same time step; style features come from VGG-19 Gram matrices and text/prompt features from Llama 3.1 embeddings.
Key results
- MTSTRec achieves NDCG@5 of 0.8800 on Food E-commerce vs 0.8512 for SASRec+ (best baseline).
- On House-Hold E-commerce MTSTRec reaches NDCG@5 0.8942, about 9.7% above SASRec+.
- On H&M (Trousers) MTSTRec reaches NDCG@5 0.2307, roughly 31.5%-43.7% above competing models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MTSTRec instead of the strongest baseline | Sequential recommendation · Food E-commerce | NDCG@5: 0.8512 -> 0.8800 | Helped |
| use MTSTRec instead of the strongest baseline | Sequential recommendation · House-Hold E-commerce | NDCG@5: 0.8401 -> 0.8942 | Helped |
| use MTSTRec instead of the strongest baseline | Sequential recommendation · H&M (Trousers) | NDCG@5: 0.1754 -> 0.2307 | Helped |
| add the item ID modality (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.7582(±0.0049) -> 0.8800(±0.0023); 0.7913(±0.0244) -> 0.8942(±0.0035) | Helped |
| add text and prompt-text modalities (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8574(±0.0013) -> 0.8800(±0.0023); 0.8191(±0.0047) -> 0.8942(±0.0035) | Helped |
| add the text modality (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8729(±0.0016) -> 0.8800(±0.0023); 0.8488(±0.0095) -> 0.8942(±0.0035) | Helped |
| add the prompt-text modality (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8749(±0.0046) -> 0.8800(±0.0023); 0.8770(±0.0035) -> 0.8942(±0.0035) | Mixed |
| add the style modality (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8784(±0.0020) -> 0.8800(±0.0023); 0.8932(±0.0093) -> 0.8942(±0.0035) | No clear effect |
| add the price modality (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8791(±0.0025) -> 0.8800(±0.0023); 0.8941(±0.0048) -> 0.8942(±0.0035) | No clear effect |
| use time-aligned TST instead of late fusion (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8211 -> 0.8800; 0.8773 -> 0.8942 | Helped |
| use time-aligned TST instead of early fusion (ablation) | MTSTRec · Food E-commerce and House-Hold E-commerce | NDCG@5: 0.8621 -> 0.8800; 0.8366 -> 0.8942 | Helped |
| use two shared tokens per time step instead of one (ablation) | MTSTRec · Food E-commerce | NDCG@5: 0.8800(±0.0023) -> 0.8795(±0.0022) | Hurt |
Surprising findings
- Removing item ID causes the largest drop (NDCG@5 0.8800 to 0.7582 on Food), while removing price barely changes results.
- Late fusion on Food drops NDCG@5 to 0.8211, lower than early fusion at 0.8621.
Limitations
- Evaluated only on three e-commerce datasets, two of which are proprietary.
- Gains over baselines are reported without variance across more datasets or external validation.
- Price contributes little to accuracy, so the price modality's value is unclear.
Experimental setup
Datasets: Food E-commerce (AviviD proprietary, released) House-Hold E-commerce (AviviD proprietary, released) H&M trousers purchases
Benchmarks: Food E-commerce House-Hold E-commerce H&M (Trousers)
Metrics: NDCG@5 NDCG@10 HR@5 HR@10 MRR@5 MRR@10
Baselines: SASRec BERT4Rec SASRec+ BERT4Rec+ MMMLP MMMLP+
Builds on: SASRec BERT4Rec attention bottleneck LLM-Rec Neural Style Transfer
Closest papers by meaning
- Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation (AAAI 2025 · similarity 0.87)
- MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation (AAAI 2025 · similarity 0.86)
- STAIR: Manipulating Collaborative and Multimodal Information for E-Commerce Recommendation (AAAI 2025 · similarity 0.86)
- Semantic Convergence: Harmonizing Recommender Systems via Two-Stage Alignment and Behavioral Semantic Tokenization (AAAI 2025 · similarity 0.86)
- LGMRec: Local and Global Graph Learning for Multimodal Recommendation (AAAI 2024 · similarity 0.86)
- Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.