Research map / Vision-language and multimodal / Human and embodied modeling
Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes uncertainty modeling and uncertainty regularization (Gaussian feature jitter on target features plus heteroscedastic-style loss weighting with annealed gamma) to unify fine- and coarse-grained matching for composed image retrieval with text feedback.
Problem
Composed image retrieval with text feedback is trained with one-to-one pair-based metric losses (InfoNCE), which push away multiple valid candidates and hurt recall at coarse-grained, one-to-many test settings.
Approach
Simulate one-to-many matching by adding identically distributed Gaussian fluctuations to target features in the representation space, and weight the InfoNCE loss by the fluctuation level (aleatoric-uncertainty-style regularization), combined with plain InfoNCE via an exponentially annealed weight.
Key results
- On FashionIQ with one ResNet-50, average Recall@50 rises from 57.23 (baseline) to 61.39, above CLIP4Cir's 59.03 (4x ResNet-50 ensemble).
- On Shoes, Recall@50 improves from 76.46 to 79.84 over the baseline; on Fashion200k, Recall@50 goes from 67.8 to 70.2.
- Adding the method to CLIP4Cir and BLIP improves Shoes Recall@50 from 70.21 to 75.29 for BLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-grained uncertainty modeling and regularization to the ResNet-50 baseline | Composed image retrieval model · FashionIQ average | R@50: 57.23 -> 61.39 | Helped |
| Add multi-grained uncertainty modeling and regularization to the ResNet-50 baseline | Composed image retrieval model · Shoes | R@50: +3.38% | Helped |
| Add multi-grained uncertainty modeling and regularization to the ResNet-50 baseline | Composed image retrieval model · Fashion200k | R@50: +2.40% | Helped |
| Use uncertainty regularization alone, without fine-grained InfoNCE (ablation) | ResNet-50 retrieval model · Shoes | R@50: 76.46% -> 77.41% | Helped |
| Add dropout instead of uncertainty regularization (ablation) | InfoNCE baseline · Shoes | R@50: 76.46 -> 76.37 (rate 0.2); 76.46 -> 75.83 (rate 0.5) | No clear effect |
| Augment source image feature f_s^I instead of target feature (ablation) | Uncertainty retrieval model · Shoes | R@50: -2.09% | Hurt |
| Set noise scales to w1=w2=1 rather than varying either scale (ablation) | Uncertainty modeling · Shoes | R@50: w1=1: 79.84; w1=0.1: 75.57, w1=0.2: 76.35, w1=0.5: 78.84, w1=0.7: 78.95, w1=2: 76.80, w1=5: 78.06, w1=7: 76.23, w1=10: 7 | Helped |
| Use dynamic balance with gamma0=1 rather than other tested initial values (ablation) | Uncertainty retrieval model · Shoes | R@50: gamma0=1: 79.84; gamma0=0.1: 78.64, 0.5: 79.12, 2: 77.63, 3: 77.55, 5: 76.32, 10: 76.43, +infinity: 76.46 | Helped |
| Replace the dynamic gamma schedule with a constant gamma (ablation) | Uncertainty retrieval model · Shoes | R@50: dynamic gamma0=1: 79.84; static gamma=0.2: 63.12, 0.5: 73.11, 0.8: 72.39 | Hurt |
| Add uncertainty regularization to the reimplemented CLIP4Cir model | CLIP4Cir with one ResNet-50 · FashionIQ | R@50: 56.74 -> 59.23 | Helped |
| Add uncertainty regularization to the BLIP backbone | BLIP · Shoes | R@50: 70.21 -> 75.29 | Helped |
Surprising findings
- Dropout at rates 0.2 and 0.5 does not improve recall, while the uncertainty regularization alone raises Recall@50 on Shoes.
- Augmenting the source image feature instead of the target hurts Recall@10 and Recall@50 on Shoes, which the authors attribute to conflicting with one-to-many matching.
Limitations
- Gains on Shoes and Fashion200k are modest relative to some baselines (ARTEMIS leads on Fashion200k R@1 and R@50).
- Evaluated only on fashion and shoe datasets; generalization to common-object retrieval is left for future work.
- Dataset download issues forced the FashionIQ training set to be partly reconstructed (75,384 images downloaded, 46,609 used), which may limit reproducibility.
Experimental setup
Datasets: FashionIQ Shoes Fashion200k
Benchmarks: FashionIQ Shoes Fashion200k
Metrics: Recall@1 Recall@10 Recall@50
Baselines: MRN FiLM TIRG Pic2Word VAL ARTEMIS CoSMo DCNet FashionViL CLVC-Net CLIP4Cir
Builds on: CosMo CLIP4Cir InfoNCE aleatoric uncertainty
Closest papers by meaning
- You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval (CVPR 2024 · similarity 0.86)
- Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval (AAAI 2024 · similarity 0.86)
- Learning with Noisy Triplet Correspondence for Composed Image Retrieval (CVPR 2025 · similarity 0.85)
- QuRe: Query-Relevant Retrieval through Hard Negative Sampling in Composed Image Retrieval (ICML 2025 · similarity 0.85)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.85)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.