Research map / Vision-language and multimodal / Human and embodied modeling
Improving Personalized Search with Regularized Low-Rank Parameter Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes POLAR, which personalizes CLIP's text encoder by learning a regularized rank-one LoRA update to the final-layer value transform per concept, and merges concept updates by addition for multi-concept retrieval.
Problem
Personalized vision-language retrieval: recognizing new personal concepts (e.g., "my dog Fido") from a few images while retaining general knowledge so concepts can be composed with context in natural language queries.
Approach
Replace textual inversion pseudo-tokens with a small rank-one low-rank update to the value transform of CLIP's final text encoder layer, regularized (L2 penalty on B and unit-norm constraint on A) so general knowledge is preserved, without needing extra training data or large-scale pretrained components.
Key results
- On DeepFashion2 (ViT-B/32, 5 images), context mRR 34.82 vs. PALAVRA 28.4 and SEARLE 21.90.
- On ConCon-Chi (ViT-L/14), context mRR 46.33 and mAP 32.33 vs. SEARLE 43.93 and 30.74; concept-only mRR 100.00 vs. 96.67.
- Adding LoRA updates for multi-concept queries (mRR 35.13) outperforms orthogonal adaptation (28.38) on ConCon-Chi.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace textual inversion with a regularized rank-1 update to the final-layer value transform | CLIP text encoder · DeepFashion2, ViT-B/32 | context mRR: 28.4 -> 34.82 | Helped |
| replace textual inversion with a regularized rank-1 update to the final-layer value transform | CLIP text encoder · ConCon-Chi | context mRR: 43.93 -> 46.33 | Helped |
| increase LoRA rank from 1 to 2 (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 51.64 -> 52.31 | Mixed |
| increase LoRA rank from 1 to 4, 8, or 16 (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 51.64 -> 51.52; 51.64 -> 51.49; 51.64 -> 51.67 | No clear effect |
| apply LoRA to layers earlier than layer 12 (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 51.64 -> 50.18; 51.64 -> 51.51; 51.64 -> 43.23; 51.64 -> 44.69 | Hurt |
| update the value transform rather than Q, K, O, MLP, or final projection (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 16.65; 15.55; 46.99; 43.04; 49.40; 51.15 -> 51.64 | Helped |
| add Q or Q,K,V,O updates alongside the value update (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 51.64 -> 51.50; 51.64 -> 47.52 | No clear effect |
| remove both the unit-norm constraint on A and the L2 penalty on B (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 51.64 -> 22.51 | Hurt |
| add both the unit-norm constraint on A and the L2 penalty on B (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 22.51 -> 51.64 | Helped |
| add the unit-norm constraint on A (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 22.51 -> 33.77 | Helped |
| add the L2 penalty on B (ablation) | POLAR · ConCon-Chi single-concept context queries | mRR: 22.51 -> 39.84 | Helped |
| add learned concept LoRA updates instead of averaging, max-pooling, or orthogonal adaptation (ablation) | POLAR multi-concept merging · ConCon-Chi multi-concept queries | mRR: 25.03; 25.27; 28.38 -> 35.13 | Helped |
Surprising findings
- Updating CLIP's Q or K transforms alone performs very poorly for personalization, while the value transform in the final layer works best.
- Orthogonal adaptation, effective in diffusion personalization, underperforms here because a random fixed A can zero out the update.
Limitations
- Retrieval-only setting; evaluated on relatively small personal-concept benchmarks (DeepFashion2 and ConCon-Chi).
- General knowledge retention measured with LLaVA-generated captions as a proxy metric.
Experimental setup
Datasets: DeepFashion2 ConCon-Chi
Benchmarks: DeepFashion2 ConCon-Chi
Metrics: mRR recall@k mAP VLM caption recall@10
Baselines: PALAVRA SEARLE Pic2Word COLLIE Textual Inversion Prompt Tuning Orthogonal Adaptation
Builds on: LoRA PALAVRA Perfusion Orthogonal Adaptation textual inversion
Closest papers by meaning
- RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models (CVPR 2025 · similarity 0.87)
- Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval (AAAI 2025 · similarity 0.87)
- Ego: Embedding-Guided Personalization of Vision-Language Models (CVPR 2026 · similarity 0.87)
- ConCon-Chi: Concept-Context Chimera Benchmark for Personalized Vision-Language Tasks (CVPR 2024 · similarity 0.87)
- Contextualized Visual Personalization in Vision-Language Models (ICML 2026 · similarity 0.86)
- EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.