Research map / Vision-language and multimodal / Cross-modal learning
Dynamic Weighted Combiner for Mixed-Modal Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Weighted Combiner (DWC) for mixed-modal image retrieval, combining editable modality de-equalizer, soft-similarity label generation, and a CLIP-based mutual-enhancement contrastive training, reaching SOTA on Fashion200k, Shoes, and FashionIQ.
Problem
Mixed-modal image retrieval (image+text query) performs poorly because image and text modalities contribute unequally, and web text labels are noisy and biased across annotators.
Approach
Edit/purify image and text features with cross-modal spatial/word attention and global attention, adaptively weight the modalities, relax hard one-hot negatives into dynamic soft similarity labels, and mutually distill between a CNN stream and a CLIP stream trained with a mixed-modality contrastive loss.
Key results
- On Fashion200k, DWC reaches R@1 36.49% and average 59.70%, improving R@1 by 12.49% over the second-best method.
- On FashionIQ, DWC achieves average mean 48.75% and best R@10 average 36.11%, slightly below FashionVLP on R@50 (61.39 vs 62.51).
- On Shoes, DWC attains R@1 18.94% and average 51.56%, improving over the prior best by 0.83% R@1 and 1.06% average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DWC instead of the strongest listed baseline | FashionIQ retrieval · FashionIQ Average | R@10: 34.27 -> 36.11 | Helped |
| Use DWC instead of GA | Fashion200K retrieval · Fashion200K | R@1: 24.00 -> 36.49 | Helped |
| Use DWC instead of CLVC-Net | Shoes retrieval · Shoes | R@10: 54.40 -> 55.55 | Helped |
| Add EMD to DWC (ablation) | DWC · FashionIQ Dress | R@10: 23.55 -> 32.67 | Helped |
| Add SSG to DWC (ablation) | DWC · FashionIQ Shirt | R@10: 34.99 -> 35.53 | Helped |
| Add the CLIP stream to DWC (ablation) | DWC · FashionIQ Shirt | R@10: 24.48 -> 35.53 | Helped |
| Add mutual enhancement (ablation) | DWC · FashionIQ Shirt | R@10: 27.82 -> 35.53 | Helped |
| Add the hard-label mixed-modality contrastive loss (ablation) | DWC · FashionIQ Dress | R@10: 31.82 -> 32.67 | Helped |
| Add the soft-label mixed-modality contrastive loss (ablation) | DWC · FashionIQ Dress | R@10: 31.73 -> 32.67 | Helped |
| Add distillation loss (ablation) | DWC · FashionIQ Dress | R@10: 29.74 -> 32.67 | Helped |
| Add the text feature editor (ablation) | EMD · FashionIQ Dress | R@10: 30.49 -> 32.67 | Helped |
| Add the image feature editor (ablation) | EMD · FashionIQ Dress | R@10: 28.01 -> 32.67 | Helped |
Surprising findings
- Naive direct weighting of image and text features hardly improves over simple concatenation, indicating that amplifying the dominant modality's errors hurts performance.
Limitations
- Failure cases when text is incomplete or unrelated to the query image, making modality importance hard to predict.
- Slightly lower R@50 than FashionVLP on FashionIQ, which uses auxiliary object detection and landmark modules.
Experimental setup
Datasets: Fashion200k Shoes FashionIQ
Benchmarks: Fashion200k Shoes FashionIQ
Metrics: R@1 R@10 R@50 mean recall
Baselines: TIRG VAL ComposeAE JVSM SynthTripletGAN CoSMo JPM DATIR CLVC-Net ARTEMIS SAC GA
Builds on: TIRG CLIP Combiner deep mutual learning InfoNCE
Closest papers by meaning
- Data Roaming and Quality Assessment for Composed Image Retrieval (AAAI 2024 · similarity 0.86)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.86)
- DiCA: Disambiguated Contrastive Alignment for Cross-Modal Retrieval with Partial Labels (AAAI 2025 · similarity 0.85)
- Do Composed Image Retrieval Benchmarks Require Multimodal Composition? (NeurIPS 2026 · similarity 0.85)
- Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation (NeurIPS 2025 · similarity 0.85)
- Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.