Research map / Vision-language and multimodal / Human and embodied modeling
Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Delta Generator (VDG), an LLaVA/InstructBLIP-style LLM fine-tuned with LoRA to describe visual differences between image pairs, used to generate pseudo triplets from auxiliary galleries for semi-supervised composed image retrieval.
Problem
Composed image retrieval needs costly labeled (reference image, visual delta, target image) triplets, while zero-shot methods trained on image-caption pairs perform notably worse. The paper asks how to exploit unlabeled image collections to augment CIR training.
Approach
Fine-tune a multimodal LLM (InstructBLIP-style Q-Former vision projector plus LLaMA2-13B with LoRA) to generate textual visual deltas between two images; then mine visually similar reference-target pairs from unlabeled galleries and label them with VDG to create pseudo triplets. The approach is model-agnostic and is paired with a target-delta matching loss for BLIP-based CIR.
Key results
- On CIRR test, BLIP_tdm trained with VDG visual deltas on NLVR2' auxiliary gallery reaches R@1 50.96 in the semi-supervised setting, above the 49.35 of CASE with external pretraining.
- In the unseen (no human triplets) setting on CIRR test, BLIP_tdm by NLVR2' achieves R@1 45.74 versus 38.48 for zero-shot CoVR.
- Replacing human-annotated deltas with VDG deltas yields only a marginal drop (BLIP_tdm R@1 52.69 vs. 53.17 human on CIRR validation), and VDG training data (e) gives higher R@5/R@10 on the human validation set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| augment supervised CIR training with VDG-generated visual deltas | BLIPtdm · CIRR test | R@1: 48.94 -> 49.08 | Helped |
| add NLVR2 auxiliary-gallery pseudo triplets | BLIPtdm · CIRR test | R@1: 48.94 -> 50.96 | Helped |
| augment supervised CIR training with VDG-generated visual deltas | BLIPtdm · FashionIQ validation | R@10: 44.87 -> 46.90; 49.61 -> 50.28; 50.54 -> 52.73 | Helped |
| add FashionIQ auxiliary-gallery pseudo triplets | BLIP†tdm · FashionIQ validation | R@10: 46.90 -> 47.89; 50.28 -> 51.36 | Helped |
| replace human-annotated deltas with VDG-generated deltas | CIR model · CIRR validation | R@1: 37.98 -> 35.47; 53.17 -> 50.16 | Hurt |
| combine human-annotated and VDG-generated deltas | CIR model · CIRR validation | R@1: 37.98 -> 39.11; 53.17 -> 53.67 | Helped |
| use similarity-based image grouping (ablation) | BLIPtdm semi-supervised CIR · CIRR validation | R@1: 52.45 -> 55.66 | Helped |
| concatenate supervised and pseudo batches (ablation) | BLIPtdm semi-supervised CIR · CIRR validation | R@1: 55.66 -> 56.28 | Helped |
| add target-delta matching loss (ablation) | BLIPtdm semi-supervised CIR · CIRR validation | R@1: 56.28 -> 57.88 | Helped |
| increase auxiliary-gallery data scale (ablation) | BLIPtdm by COCO′ · CIRR test | R@1: 42.15 -> 43.49 | Helped |
| increase auxiliary-gallery data scale (ablation) | BLIPtdm by NLVR2′ · CIRR test | R@1: 44.31 -> 45.74 | Helped |
Surprising findings
- VDG-generated deltas used for training on CIRR validation yielded higher R@5 and R@10 than human-annotated deltas, which the authors attribute to inconsistency in human annotation styles.
Limitations
- Semi-supervised gains saturate once pseudo-selected images approximate the training set size, possibly because pseudo triplets fail to match test distribution.
- Relies on a 13B LLM plus ViT-G and Q-Former, making VDG labeling computationally heavy.
- Pseudo triplets require visually similar image subgroups, and overlap with test sets must be filtered manually.
Experimental setup
Datasets: CIRR FashionIQ CC3M NLVR2 COCO DeepFashion
Benchmarks: CIRR FashionIQ
Metrics: R@1 R@5 R@10 R@50 Rs@1 Rs@2 Rs@3
Baselines: Combiner BLIP ARTEMIS CIRPLANT CASE CoVR Pic2Word SEARLE
Builds on: InstructBLIP LLaVA CLIP BLIP Combiner
Closest papers by meaning
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.93)
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.91)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.91)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.91)
- Language-only Training of Zero-shot Composed Image Retrieval (CVPR 2024 · similarity 0.90)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.