Research map / Vision-language and multimodal / Human and embodied modeling
Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIR-LVLM, which fine-tunes Qwen-VL-Chat as a single-pass user intent-aware encoder for composed image retrieval, using task prompts and a learnable instance-specific soft prompt pool.
Problem
Composed image retrieval needs both comprehensive extraction of reference-image visual information and faithful understanding of the user's modification intent in the relative caption; existing VLPM fusion and multi-pass generative LLM approaches fall short on accuracy or efficiency.
Approach
Use a large vision-language model (Qwen-VL-Chat) as the encoder for hybrid-modality queries and target images, guided by a two-level hybrid intent instruction: a hand-written task prompt plus an instance-specific soft prompt selected from a learnable key-value prompt pool via image and text query matching; embeddings are position-weighted mean-pooled and trained with a batch contrastive loss.
Key results
- On Fashion-IQ, CIR-LVLM reaches R_mean 66.17 average, versus 64.85 for SPRC and 62.20 for TG-CIR (the paper reports a 3.97% gain over TG-CIR).
- On Shoes, R_mean 63.51 versus 58.05 for TG-CIR (5.46% improvement reported).
- On CIRR, average of Recall@5 and Recall_subset@1 is 81.44, slightly above SPRC's 81.38; it trails SPRC on Recall_subset@1 (79.12 vs 80.65).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CIR-LVLM instead of SPRC | CIR retrieval on Fashion-IQ · Fashion-IQ | R_mean: 64.85 -> 66.17 | Helped |
| use CIR-LVLM instead of TG-CIR | CIR retrieval on Shoes · Shoes | R_mean: 58.05 -> 63.51 | Helped |
| use CIR-LVLM instead of SPRC | CIR retrieval on CIRR · CIRR | Avg.: 81.38 -> 81.44 | Helped |
| use a fine-tuned Qwen-VL-Chat encoder instead of CLIP or BLIP-2 text encoder (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 45.34 -> 66.17; 54.85 -> 66.17 | Helped |
| add a brief task prompt instead of using no prompt (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 63.84 -> 65.04 | Helped |
| add a detailed task prompt instead of using no prompt (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 63.84 -> 65.50 | Helped |
| use instance-specific soft prompts instead of a universal prompt (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 64.95 -> 66.17 | Helped |
| fine-tune with LoRA instead of full fine-tuning (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 59.34 -> 66.17 | Helped |
| add Connector fine-tuning to LoRA (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 64.09 -> 66.17 | Helped |
| use Qwen-VL-Chat instead of LLaVA-1.5 or LLaVA-1.6 as backbone (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 61.63 -> 66.17; 62.65 -> 66.17 | Helped |
| use weighted mean pooling instead of last-token pooling (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 65.70 -> 66.17 | Helped |
| use weighted mean pooling instead of regular mean pooling (ablation) | CIR-LVLM · Fashion-IQ | R_mean: 65.61 -> 66.17 | Helped |
Surprising findings
- Replacing the LLM with CLIP or BLIP-2 text encoders markedly lowers performance, suggesting the LVLM's reasoning helps intent understanding.
- Full fine-tuning underperforms LoRA by about 6% R_mean on Fashion-IQ, suggesting overfitting.
Limitations
- Lags SPRC on CIRR Recall_subset@1, struggling with targets semantically or visually similar to the reference image.
- Failure cases with short relative captions cause over-retention of reference image information.
- Inference is slower than lightweight VLPM encoders (0.08s vs 0.035s per query).
Experimental setup
Datasets: Fashion-IQ Shoes CIRR
Benchmarks: Fashion-IQ Shoes CIRR
Metrics: Recall@K R_mean Recall_subset@K
Baselines: TIRG CLIP4CIR BLIP4CIR CASE TG-CIR Re-ranking SPRC CIReVL CoVR-BLIP ARTEMIS
Builds on: L2P BLIP Qwen-VL LoRA CLIP4CIR
Closest papers by meaning
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.90)
- Sentence-level Prompts Benefit Composed Image Retrieval (ICLR 2024 · similarity 0.89)
- Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.88)
- Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.