Research map / Vision-language and multimodal / Multimodal LLMs
Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLLM-MSR, which converts item images to text with an MLLM, recurrently summarizes user preferences block-by-block with an LLM, and fine-tunes an MLLM recommender with LoRA-based SFT for multimodal sequential recommendation.
Problem
Multimodal sequential recommendation with MLLMs is underexplored: processing many ordered image inputs is costly, MLLMs struggle with temporal user dynamics, and fine-tuning risks overfitting and losing generalization.
Approach
Summarize item images into text with an MLLM, then infer user preferences recurrently over session blocks with an LLM summarizer, and use the resulting preference summaries plus item text and image in a yes/no SFT-tuned MLLM recommender whose score is p(yes)/(p(yes)+p(no)).
Key results
- MLLM-MSR reports AUC 83.17 on Microlens, 84.39 on Video-Games, and 85.69 on Amazon-Baby, versus 81.25, 81.79, and 82.16 for the best baseline TALLRec.
- HR@5 on Microlens improves from 71.23 (TALLRec) to 78.23 (MLLM-MSR).
- Ablation: removing recurrent preference inference (MLLM-MSR_R) and removing image summaries (MLLM-MSR_I) both reduce performance relative to the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Recurrent user preference inference (vs. direct inference over full history) (ablation) | MLLM-MSR · ablation (three datasets) | AUC/HR@5/MRR@5 | Helped |
| Removing image summarization from item summarization (text-only preference inference) (ablation) | MLLM-MSR · ablation (three datasets) | AUC/HR@5/MRR@5 | Helped |
| Image summary (LLaVA text summary) instead of VGG19 visual features in GRU4Rec | GRU4Rec · Microlens preliminary | AUC: +0.0127 on Microlens; -0.0065 on Baby; -0.0081 on Games | Mixed |
| Fine-tuned MLLM recommender (SFT with LoRA) vs. LLaVA without SFT | LLaVA-v1.6-mistral-7b · Microlens | AUC: +30.42 AUC (83.17 vs 52.75) | Helped |
| Text-only LLM SFT recommender (TALLREC) vs. proposed multimodal MLLM-MSR | MLLM-MSR · Microlens, Video-Games, Amazon-Baby | AUC: +1.92 / +2.60 / +3.53 AUC over TALLREC | Helped |
Surprising findings
- In the preliminary study, GRU4Rec with image summaries performs comparably to VGG19 visual features, and is even lower on Microlens (0.7281 vs 0.7154 is higher; but lower on Baby and Games).
Limitations
- Results are reported without clear variance or per-setting details despite claimed 5-fold cross-validation and 95% confidence level.
- Dataset naming is inconsistent (Video-Games vs Amazon-Game) and the preliminary study uses different numbers than the main table.
- Relies on a large 7B-8B MLLM and LLM pipeline with heavy per-item summarization, raising compute cost.
Experimental setup
Datasets: MicroLens Amazon-Baby Amazon-Game
Benchmarks: MicroLens Video-Games Amazon-Baby
Metrics: AUC HR@5 MRR@5
Baselines: GRU4Rec SASRec MGAT MMGCN GRU4Rec-F SASRec-F Trans2D MMSR LLaVA w/o SFT TALLRec
Builds on: TALLRec LLaVA GRU4Rec SASRec
Closest papers by meaning
- Multimodal Quantitative Language for Generative Recommendation (ICLR 2025 · similarity 0.89)
- SLMRec: Distilling Large Language Models into Small for Sequential Recommendation (ICLR 2025 · similarity 0.89)
- LLM-ESR: Large Language Models Enhancement for Long-tailed Sequential Recommendation (NeurIPS 2024 · similarity 0.89)
- DOGE: LLMs-Enhanced Hyper-Knowledge Graph Recommender for Multimodal Recommendation (AAAI 2025 · similarity 0.88)
- Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation (NeurIPS 2025 · similarity 0.88)
- MTSTRec: Multimodal Time-Aligned Shared Token Recommender (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.