Research map / Image generation / Autoregressive and latent generation
Decomposing Semantic Shifts for Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Semantic Shift Network (SSN) that treats modification text as an instruction, decomposing it into degradation and upgradation steps that map reference image → visual prototype → target image for composed image retrieval.
Problem
Composed image retrieval systems treat the modification text as a description and fuse it with the reference image, which ignores the text's instruction structure and leads to shortcuts that discard reference-image visual cues.
Approach
Reinterpret the text as an instruction that encodes a semantic shift; decompose it into positive (L+) and negative (L-) guidance via learned sigmoid token gates, degrade the reference image into a visual prototype using L-, then upgrade the prototype with L+ through a transformer fusion and a KL-based regularizer.
Key results
- On CIRR test, SSN reaches mean recall 74.51 vs CLIP4Cir 69.09, a 5.42-point gain; Recall@1 43.91 vs 38.53.
- On FashionIQ, SSN achieves average mean recall 51.40 vs CLIP4Cir 50.03, a 1.37-point gain.
- Ablation on CIRR validation: full SSN improves Recall@1 by 2.51 points over the baseline (45.13 vs 42.62).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic-shift decomposition in SSN instead of CLIP4Cir | CIRR · CIRR test | (R@5+R_sub@1)/2: 5.42% | Helped |
| Use semantic-shift decomposition in SSN instead of the strongest baseline | FashionIQ · FashionIQ validation | mean recall: 1.37% | Helped |
| Add semantic-shift decomposition to the baseline (ablation) | CIRR validation · CIRR validation | Recall@1: 42.62 -> 45.13 | Helped |
| Remove degradation and use original reference and text features (ablation) | SSN · CIRR validation | Recall@1: 45.13 -> 43.34 | Hurt |
| Use visual prototypes with original text features (ablation) | SSN · CIRR validation | Recall@1: 45.13 -> 44.43 | Hurt |
| Use original reference features with positive guiding text (ablation) | SSN · CIRR validation | Recall@1: 45.13 -> 43.46 | Hurt |
| Add fine-grained token features without degradation (ablation) | CIRR baseline · CIRR validation | Recall@1: 42.62 -> 43.34; 0.72% | Helped |
| Add visual-prototype generation to the model with original features (ablation) | SSN(I_r,L) · CIRR validation | Recall@1: up to 1.79% | Helped |
| Add regularization loss L_k (ablation) | SSN · CIRR validation | mean: 74.67 -> 75.27 | Helped |
| Share parameters between image and text decomposing layers (ablation) | SSN · CIRR validation | mean: 75.27 -> 74.91 | Hurt |
| Add scaled Gaussian noise to reference-image features (ablation) | SSN versus baseline · CIRR validation | Recall@1: Baseline: 42.62 -> 28.25; SSN: 45.13 -> 26.12 | Mixed |
Surprising findings
- Sharing the decomposing MLP parameters between image and text branches hurts performance, since the two branches have different objectives.
- Adding Gaussian noise to reference image features degrades SSN more than the baseline (CIRR mean 57.90 vs 72.84 baseline reference-noise comparison), which the authors read as evidence the model relies on reference visual prototypes.
Limitations
- Gains on FashionIQ are smaller (1.37 points), attributed to domain gap and small dataset size.
- Model is not designed for zero-shot settings; authors list zero-shot extension as future work.
Experimental setup
Datasets: CIRR FashionIQ
Benchmarks: CIRR FashionIQ
Metrics: Recall@K Recall_subset@K mean recall (R@5+R_sub@1)/2 R@10 R@50 average
Baselines: TIRG MAAF ARTEMIS CIRPLANT CLIP4Cir VAL CoSMo CLVC-Net SAC DCNet MUR JVSM
Builds on: CLIP4Cir CLIP TIRG BERT
Closest papers by meaning
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.86)
- FashionERN: Enhance-and-Refine Network for Composed Fashion Image Retrieval (AAAI 2024 · similarity 0.86)
- Generative Zero-Shot Composed Image Retrieval (CVPR 2025 · similarity 0.85)
- Data Roaming and Quality Assessment for Composed Image Retrieval (AAAI 2024 · similarity 0.85)
- Sentence-level Prompts Benefit Composed Image Retrieval (ICLR 2024 · similarity 0.85)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.