Research map / Vision-language and multimodal / Cross-modal learning
Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniMP, a unified multi-modal personalization framework that feeds interleaved image-text user histories into a vision-augmented LLM (INCITE-3B) and casts recommendation, search, preference prediction, explanation, selection and user-guided image generation as instruction-tuned generation tasks.
Problem
Existing personalization systems handle a single task and a single modality (ID or text), and cannot process heterogeneous multi-modal user histories (images, text, IDs, tabular attributes) or jointly serve multiple personalization tasks.
Approach
A unified key-value attribute input format with [IMG]/[EOC] tokens, layer-wise gated cross-attention with exclusive per-item masking to fuse item images into a frozen LM, multi-task instruction tuning with context reconstruction loss and token-level focal re-weighting.
Key results
- On Amazon product recommendation, UniMP reaches HR@5 0.0337 and NDCG@5 0.0231, versus 0.0281/0.0194 for the best multi-modal baseline MMSSL and 0.0262/0.0163 for VIP5+.
- Personalized multi-modal search HR@5 0.1769 vs 0.0981 for RTM; preference prediction MAE 0.7049 vs 0.9122 for SANCL.
- Removing vision information drops recommendation HR@5 from 0.0337 to 0.0284.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add vision information to user modeling (ablation) | UniMP · Amazon recommendation | HR@5: 0.0284 -> 0.0337 | Helped |
| Add layer-wise cross-attention for fine-grained visual-text fusion (ablation) | UniMP · Amazon recommendation | HR@5: 0.0257 -> 0.0337 | Helped |
| Add context reconstruction and token-level re-weighting (ablation) | UniMP · Amazon recommendation | HR@5: 0.0311 -> 0.0337 | Helped |
| Add context reconstruction (ablation) | UniMP · Amazon recommendation | HR@5: 0.0318 -> 0.0337 | Helped |
| Add token-level re-weighting (ablation) | UniMP · Amazon recommendation | HR@5: 0.0302 -> 0.0337 | Helped |
| Use exclusive attention between each product's text and visual information (ablation) | UniMP · Amazon recommendation | HR@5: 0.0244 -> 0.0337 | Helped |
| Use cross-attention rather than no cross-attention (ablation) | UniMP · Amazon recommendation | HR@5: 0.0257 -> 0.0337 | Helped |
| Use the proposed fusion rather than early or late fusion (ablation) | UniMP · Amazon recommendation | HR@5: 0.0271 -> 0.0337; 0.0274 -> 0.0337 | Helped |
| Remove 20% of textual attribute information (ablation) | UniMP · Amazon recommendation with missing text attributes | HR@5: 0.0337 -> 0.0332 | Hurt |
| Mismatch 20% of images with their products (ablation) | UniMP · Amazon recommendation with mismatched images | HR@5: 0.0337 -> 0.0328 | Hurt |
| Use joint multi-task learning instead of continuous task learning (ablation) | UniMP · Amazon recommendation | HR@5: 0.0197 -> 0.0293 | Helped |
| Fine-tune only the last layer instead of the full model after multi-task learning (ablation) | UniMP · Amazon recommendation | HR@5: 0.0325 -> 0.0337 | Helped |
Surprising findings
- Fine-tuning only the last layer (Multi-task+Efficient) outperformed full-model fine-tuning on recommendation and search.
- Replacing the 3B LM with the 1B MPT model reduces performance only modestly (HR@5 0.0315 vs 0.0337).
Limitations
- Evaluated only on Amazon-style e-commerce and two other datasets; no video or audio modalities.
- Baseline comparisons are limited to a single 3B LM family; generalization claims rest on one held-out domain (Office).
- Code and weights promised only after review, not released in the paper.
Experimental setup
Datasets: Amazon Baby Amazon Beauty Amazon Clothing Amazon Grocery Amazon Sports Amazon Toys Amazon Office HM Netflix
Benchmarks: Amazon multi-domain recommendation benchmark personalized explanation generation preference prediction multi-modal selection multi-modal search HM Netflix
Metrics: HR@5 NDCG@5 MRR@5 BLEU ROUGE METEOR BERTScore MAE RMSE Recall
Baselines: SASRec S3-Rec UniSRec P5 VIP5 MMSSL CausalRec LightGCN
Builds on: Flamingo P5 VIP5 OpenFlamingo focal loss
Closest papers by meaning
- MMPB: It’s Time for Multi-Modal Personalization (NeurIPS 2025 · similarity 0.89)
- VisualLens: Personalization through Task-Agnostic Visual History (NeurIPS 2025 · similarity 0.87)
- Multimodal Quantitative Language for Generative Recommendation (ICLR 2025 · similarity 0.87)
- Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation (AAAI 2025 · similarity 0.86)
- Preference Diffusion for Recommendation (ICLR 2025 · similarity 0.86)
- MTSTRec: Multimodal Time-Aligned Shared Token Recommender (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.