Research map / Vision-language and multimodal / Multimodal LLMs
Personalized Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PVIT, a data-curation and LoRA instruction-tuning framework that teaches LLaVA-7B to answer personalized questions about named individuals in images via multimodal in-context prefixes, plus the PVIT-3M dataset and P-Bench benchmark.
Problem
Multimodal LLMs can chat generally but fail at personalized dialogue about specific individuals ("face blindness"); existing personalization methods need per-individual training.
Approach
Treat personalization as in-context learning: individuals are given as <personal image, introduction> multimodal prefixes wrapped by special person_start/person_end tokens, and the MLLM is instruction-tuned on automatically synthesized personalized QA data (with adversarial unanswerable cases) so it generalizes to new individuals without per-person training.
Key results
- P-LLaVA reaches 96.69% average accuracy on P-Bench MC answerable questions vs. 80.71% for the best baseline (LLaVA-OneVision-7B).
- On unanswerable MC questions, P-LLaVA achieves 99.72% average vs. at most 31.49% for baselines (InternVL-Chat-V1.5-26B).
- Ablation: removing adversarial samples drops unanswerable accuracy from 99.78% to 1.12%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA with PVIT | LLaVA-7B · P-Bench MC answerable Avg | accuracy: 80.71 -> 96.69 | Helped |
| Fine-tune LLaVA with PVIT | LLaVA-7B · P-Bench MC unanswerable Avg | accuracy: 31.49 -> 99.72 | Helped |
| Fine-tune LLaVA with PVIT | LLaVA-7B · P-Bench personalized description answerable Avg | LongClip similarity: 78.13 -> 82.55 | Helped |
| Fine-tune LLaVA with PVIT | LLaVA-7B · P-Bench personalized description unanswerable Avg | rejection rate: 17.50 -> 100.00 | Helped |
| Add face augmentation (ablation) | PVIT training data · P-Bench MC Crop | accuracy: 94.93 -> 96.43 | Helped |
| Add face augmentation (ablation) | PVIT training data · P-Bench MC Augment | accuracy: 88.43 -> 95.32 | Helped |
| Add face augmentation (ablation) | PVIT training data · P-Bench MC Unanswerable | accuracy: 98.73 -> 99.78 | Helped |
| Add adversarial samples (ablation) | PVIT training data · P-Bench MC Crop | accuracy: 95.92 -> 96.43 | Helped |
| Add adversarial samples (ablation) | PVIT training data · P-Bench MC Augment | accuracy: 92.48 -> 95.32 | Helped |
| Add adversarial samples (ablation) | PVIT training data · P-Bench MC Unanswerable | accuracy: 1.12 -> 99.78 | Helped |
Surprising findings
- Without adversarial samples the model almost never rejects unanswerable queries, indicating baseline MLLMs answer by mistakenly assigning the person to others in the scene.
Experimental setup
Datasets: PVIT-3M
Benchmarks: P-Bench
Metrics: accuracy LongCLIP similarity rejection rate
Baselines: Qwen-VL-7B VILA1.5-7B LLaVA-OneVision-7B InternVL-Chat-V1.5-26B Deepseek-VL-7b-chat mPLUG-OWl2
Builds on: LLaVA PhotoMaker GroundingDINO MyVLM Yo'LLaVA
Closest papers by meaning
- MMPB: It’s Time for Multi-Modal Personalization (NeurIPS 2025 · similarity 0.89)
- MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation (AAAI 2026 · similarity 0.89)
- Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.88)
- Yo'Chameleon: Personalized Vision and Language Generation (CVPR 2025 · similarity 0.88)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.88)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.