Research map / Vision-language and multimodal / Multimodal LLMs
VisualLens: Personalization through Task-Agnostic Visual History
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisualLens, which turns a user's task-agnostic photo history into spectrum profiles (image, caption, aspect words), retrieves query-relevant photos into a gridified image, and jointly trains an MLLM for aspect generation and candidate matching on two new benchmarks (Google Review-V, Yelp-V).
Problem
Personalized recommendation usually relies on item-based interaction logs or text, which are often unavailable; the paper asks whether a user's diverse, task-agnostic visual history (photos) can support general recommendation QA with MLLMs under limited context windows.
Approach
Represent each history image as a spectrum triplet (raw image, caption, aspect words), retrieve query-relevant images via category-embedding similarity, pack up to 64 images into an 8x8 grid so a MLLM can process them within its context limit, and jointly train aspect-word refinement and candidate matching with LoRA.
Key results
- VisualLens (MiniCPM-V2.5 8B) reaches Hit@10 of 82.3 on Google Review-V and 91.0 on Yelp-V, versus 73.0 and 85.0 for UniMP.
- VisualLens (MiniCPM-V2.5 8B) improves Hit@3 over GPT-4o by about 1.6 points on Google Review-V (38.9 vs 37.3) and 4.6 points on Yelp-V (59.1 vs 54.5), per the reported table.
- Removing history retrieval reduces Hit@3 by about 7% on Google Review-V and 12% on Yelp-V with PaliGemma.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VisualLens with MiniCPM-V2.5 instead of GPT-4o | task-agnostic visual-history recommendation · Google Review-V | Hit@3: 37.3 -> 38.9 | Helped |
| Use VisualLens with MiniCPM-V2.5 instead of GPT-4o | task-agnostic visual-history recommendation · Yelp-V | Hit@3: 54.5 -> 59.1 | Helped |
| Add joint aspect-refinement and candidate-matching training (ablation) | VisualLens with PaliGemma · Google Review-V | Hit@3: 35.8 -> 36.3 | Helped |
| Add iterative aspect refinement and joint training (ablation) | VisualLens with PaliGemma · Google Review-V | Hit@3: 35.2 -> 36.3 | Helped |
| Add joint aspect-refinement and candidate-matching training (ablation) | VisualLens with PaliGemma · Yelp-V | Hit@3: 57.9 -> 58.8 | Helped |
| Add iterative aspect refinement and joint training (ablation) | VisualLens with PaliGemma · Yelp-V | Hit@3: 57.5 -> 58.8 | Helped |
| Add captions (ablation) | VisualLens with PaliGemma · Google Review-V | Hit@3: 32.5 -> 34.7 | Helped |
| Add aspect words (ablation) | VisualLens with PaliGemma · Google Review-V | Hit@3: 32.5 -> 33.9 | Helped |
| Add captions and aspect words (ablation) | VisualLens with PaliGemma · Yelp-V | Hit@3: 48.2 -> 58.8 | Helped |
| Add aspect words instead of captions (ablation) | VisualLens with PaliGemma · Google Review-V | Hit@3: 33.9 -> 34.7 | Helped |
| Add aspect words instead of captions (ablation) | VisualLens with PaliGemma · Yelp-V | Hit@3: 53.9 -> 55.3 | Helped |
| Add relevant image retrieval (ablation) | VisualLens with PaliGemma · Google Review-V | Hit@3: 27.9 -> 35.2 | Helped |
Surprising findings
- Ranking candidates by overall rating performs worse than random ranking (MRR 17.7 vs 21.2 on Google Review-V), showing overall ratings do not reflect individual preferences.
- Aspect words contribute more than captions in the ablation (removing aspects hurts more than removing captions).
Limitations
- Modules are not the best available per subtask; the framework is modular with off-the-shelf components.
- Only static images are studied; no video, audio, or temporal narratives.
- Evaluation restricted to recommendation QA format on two benchmarks derived from Google Local and Yelp.
Experimental setup
Datasets: Google Review-V Yelp-V DOCCI Google Local Review Yelp
Benchmarks: Google Review-V Yelp-V
Metrics: Hit@1 Hit@3 Hit@10 MRR
Baselines: UniMP Llama-3.1-8B-Instruct PaliGemma MiniCPM-V2.5 Llama-3.1-70B-Instruct GPT-4o Random Rank by rating
Builds on: UniMP LLaVA-v1.6 MiniCPM-V2.5 PaliGemma CLIP DOCCI
Closest papers by meaning
- Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond (ICLR 2024 · similarity 0.87)
- Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation (AAAI 2025 · similarity 0.85)
- MTSTRec: Multimodal Time-Aligned Shared Token Recommender (ICML 2025 · similarity 0.85)
- DOGE: LLMs-Enhanced Hyper-Knowledge Graph Recommender for Multimodal Recommendation (AAAI 2025 · similarity 0.85)
- UxSID: Semantic-Aware User Interests Modeling for Ultra-Long Sequence (NeurIPS 2026 · similarity 0.85)
- Action-On-Item Preference Flow: A Shared Event Schema for Predictive and Generative Personalization (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.