Research map / Vision-language and multimodal / Human and embodied modeling
ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ENCODER, a composed image retrieval model that filters latent factors, mines visual entities and modification actions with learnable relation queries, and binds them via implicit modification relations before multi-scale composition.
Problem
Composed image retrieval fails to model the modification relation between visual entities in the reference image and modification actions in the text, due to irrelevant factor perturbation, vague semantic boundaries, and implicit relations.
Approach
Latent Factor Filter with threshold gating, modality-shared Learnable Relation Queries that mine entities and actions and act as a medium for entity-action binding (via orthogonality and KL-consistency losses), and a Multi-scale Composition guided by that binding.
Key results
- On FashionIQ, ENCODER reaches 51.51 R@10 and 76.95 R@50 on dress-category-level averages, with a reported 19.8% improvement over the best baseline for R@10 on FashionIQ-Avg.
- On CIRR, ENCODER achieves R@1 46.10 and Rsubset@1 76.92 (average 77.45), surpassing SSN (74.51 average).
- On Fashion200K, ENCODER obtains R@1 24.14 and R@10 55.98, improving over MGUR's R@1 of 21.80.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ENCODER instead of FAME-ViL | FashionIQ · FashionIQ (Avg) | R@10: 46.84 -> 56.13 | Helped |
| use ENCODER instead of FAME-ViL | FashionIQ · FashionIQ (Avg) | R@50: 69.75 -> 77.59 | Helped |
| use ENCODER instead of Progressive Learning | Shoes · Shoes | Avg: 55.29 -> 59.68 | Helped |
| use ENCODER instead of SSN | CIRR · CIRR | (R@5+Rsubset@1)/2: 74.51 -> 77.45 | Helped |
| use ENCODER instead of the strongest listed baseline, CRN | Fashion200K · Fashion200K | R@1: 21.80 -> 24.14 | Helped |
| add LFF to reference image and modification text (ablation) | ENCODER · FashionIQ (Avg) | R@10: 50.08 -> 56.13 | Helped |
| apply LFF to both modalities rather than the reference image only (ablation) | ENCODER · FashionIQ (Avg) | R@10: 55.08 -> 56.13 | Helped |
| apply LFF to both modalities rather than the modification text only (ablation) | ENCODER · FashionIQ (Avg) | R@10: 52.21 -> 56.13 | Helped |
| add EAB to reference image and modification text (ablation) | ENCODER · FashionIQ (Avg) | R@10: 49.06 -> 56.13 | Helped |
| apply EAB to both modalities rather than the reference image only (ablation) | ENCODER · FashionIQ (Avg) | R@10: 54.32 -> 56.13 | Helped |
| apply EAB to both modalities rather than the modification text only (ablation) | ENCODER · FashionIQ (Avg) | R@10: 53.18 -> 56.13 | Helped |
| add both LFF and EAB (ablation) | ENCODER · FashionIQ (Avg) | R@10: 48.58 -> 56.13 | Helped |
Surprising findings
- Removing the Lkl loss causes a large drop on CIRR (average falls from 59.68 to 52.95 on Shoes-style column reporting is ambiguous), suggesting target-distribution guidance matters for multimodal query learning.
Limitations
- Performance sensitive to latent factor number P and query number E, with degradation when too large.
- Relies on a fixed CLIP ViT-B/32 backbone; no larger backbone evaluated in the text.
Experimental setup
Datasets: FashionIQ Shoes Fashion200K CIRR
Benchmarks: FashionIQ Shoes Fashion200K CIRR
Metrics: Recall@k Rsubset@k average recall
Baselines: TIRG CLVC-Net CLIP4CIR BLIP4CIR SSN Progressive Learning FashionVLP CRN ComqueryFormer MGUR
Builds on: CLIP CLIP4CIR LF-CLIP
Closest papers by meaning
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.87)
- Context-I2W: Mapping Images to Context-Dependent Words for Accurate Zero-Shot Composed Image Retrieval (AAAI 2024 · similarity 0.86)
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.86)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.86)
- Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.