Research map / Vision-language and multimodal / Visual question answering
VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Post-processes composed image retrieval rankings by fine-tuning LLaMA to generate yes/no QA pairs from relative captions and fine-tuning LLaVA to verify each candidate image, then soft-reranking inconsistent candidates.
Problem
Composed image retrieval (CIR) methods often return top-ranked images that contradict the relative caption, so the correct target is ranked lower than it should be.
Approach
Treat reranking as self-verification: an LLM decomposes the relative caption into yes/no questions, a fine-tuned LVLM answers them for each retrieved candidate, and candidates whose answers mismatch the expected QA answers are pushed down via a soft rank-shift function R(p^A)=alpha*exp(-beta*p^A). The method is plug-in and works on top of any CIR model.
Key results
- On CIRR test, SPRC+VQA4CIR reaches average 83.15 vs SPRC 81.39 and Recall@1 54.00 vs 51.96.
- On Fashion-IQ validation, SPRC+VQA4CIR reaches average 65.41 vs SPRC 64.76; CLIP4CIR*+VQA4CIR reaches 56.29 vs 55.35.
- On CIRR validation, reranking top-15 candidates with CLIP4CIR improves Recall@1 from 53.94 to 56.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VQA4CIR reranking | SPRC · CIRR test | Recall@10: 89.74 -> 91.85 | Helped |
| Add VQA4CIR reranking | SPRC · Fashion-IQ validation | Avg. recall: 64.76 vs. 65.41 | Helped |
| Rerank top 70 candidates instead of none (ablation) | VQA4CIR on CIRR · CIRR validation | Average: 82.06 -> 84.33 | Helped |
| Increase candidate count from top 70 to top 150 (ablation) | VQA4CIR · CIRR validation | Average: 84.33 -> 84.14 | Hurt |
| Use pretrained LLaVA without fine-tuning instead of fine-tuned LLaVA (ablation) | VQA4CIR · CIRR validation | Average: 84.33 vs. 81.98 | Hurt |
| Use pretrained InstructBLIP without fine-tuning instead of fine-tuned LLaVA (ablation) | VQA4CIR · CIRR validation | Average: 84.33 vs. 80.59 | Hurt |
| Fine-tune LLaVA with LoRA rather than use it pretrained (ablation) | VQA4CIR · CIRR validation | Average: 81.98 -> 84.33 | Helped |
| Train the prompt only rather than freeze it and train LoRA (ablation) | LLaMA QA generation · CIRR validation | Average: 84.33 vs. 77.99 | Hurt |
| Train both prompt and LoRA rather than freeze the prompt and train LoRA (ablation) | LLaMA QA generation · CIRR validation | Average: 84.33 vs. 82.85 | Hurt |
| Use caption-wise rather than question-wise reranking (ablation) | VQA4CIR · CIRR validation | Average recall: 2.62% and 3.27% | Helped |
| Increase α to its reported optimum (ablation) | VQA4CIR · CIRR validation | Average recall: null | Mixed |
| Increase β to its reported optimum (ablation) | VQA4CIR · CIRR validation | Average recall: null | Mixed |
Surprising findings
- Pre-trained LLaVA and InstructBLIP without fine-tuning underperform the fine-tuned VQA model, showing off-the-shelf LVLMs do not generalize well to this verification task.
- Larger candidate pools (Top 150) do not always help; performance can drop versus Top 15 because the VQA model fails to filter all negatives.
Limitations
- Relies on LLM and LVLM at inference, adding substantial inference-time cost.
- Reranking gains depend on the number of candidates C; large C can hurt performance.
- GPT-4-generated QA pairs contain low-quality cases requiring manual correction.
Experimental setup
Datasets: CIRR Fashion-IQ
Benchmarks: CIRR Fashion-IQ
Metrics: Recall@K Recall_Subset@K average recall
Baselines: CLIP4CIR SPRC Re-ranking (Liu et al.) CoVR-BLIP CASE TG-CIR LF-CLIP CompoDiff
Builds on: SPRC CLIP4CIR LLaVA LoRA
Closest papers by meaning
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.89)
- Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval (CVPR 2024 · similarity 0.89)
- Separation of Powers: On Segregating Knowledge from Observation in LLM-enabled Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.88)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.88)
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.