Research map / Vision-language and multimodal / Human and embodied modeling
Transformer-Empowered Multi-Modal Item Embedding for Enhanced Image Search in E-commerce
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIEM, a dual-tower multimodal item embedding model fusing product titles and multiple images via merge attention, deployed in Shopee image search to lift clicks and orders per user.
Problem
Image-to-image retrieval for e-commerce image search overlooks semantic information and yields heavy storage overhead when products have multiple images.
Approach
A dual-tower model whose item tower fuses multiple product images and title text with a merge attention transformer into a single item embedding, trained with AM-InfoNCE, a modality balance loss, and XBM, then combined with I2I recall.
Key results
- Online A/B test in Shopee Indonesia: +9.90% clicks per user and +4.23% orders per user for MIEM+I2I vs I2I.
- Offline on 3M-item test set: MIEM+I2I Recall@1 0.872 vs I2I 0.820; Recall@5 0.953 vs 0.922.
- ES index storage about 61.17% less than the previous I2I index.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace image-only retrieval with MITEM | Shopee image search · Shopee product test set (3 million items) | Recall@1: 0.820 -> 0.812 | Hurt |
| Replace image-only retrieval with MITEM | Shopee image search · Shopee product test set (3 million items) | Recall@5: 0.922 -> 0.927 | Helped |
| Replace I2I with CLIP image-to-text retrieval | Shopee image search · Shopee product test set (3 million items) | Category Accuracy: 0.874 -> 0.876 | Helped |
| Replace I2I with CLIP image-to-text retrieval | Shopee image search · Shopee product test set (3 million items) | Recall@1: 0.820 -> 0.395 | Hurt |
| Generate one embedding for each image-text pair (ablation) | MITEM · Shopee product test set (3 million items) | Category Accuracy: 0.881 -> 0.894 | Helped |
| Generate one embedding for each image-text pair (ablation) | MITEM · Shopee product test set (3 million items) | Recall@5: 0.927 -> 0.939 | Helped |
| Combine I2I with MITEM recall | Shopee image search · Shopee product test set (3 million items) | Recall@1: 0.820 -> 0.872 | Helped |
| Combine I2I with MITEM recall | Shopee image search · Shopee Indonesia A/B test | Clicks/User: +9.90% | Helped |
| Combine I2I with MITEM recall | Shopee image search · Shopee Indonesia A/B test | Orders/User: +4.23% | Helped |
| Combine I2I with MITEM recall | Shopee image search · Shopee Indonesia online evaluation | Same Item Recall@4: 50.55% -> 53.66% | Helped |
Surprising findings
- Text-only CLIP I2T retrieval has much lower Recall@1 (0.395) than image-to-image despite similar category accuracy, attributed to ambiguity of text descriptions.
Limitations
- Products with many SKUs may be hard to cover with a single embedding.
- Newly released products unseen in training are hard to link between text and image; requires regular retraining.
- About 5% of click triplets are unrelated noise that was not cleaned.
Experimental setup
Datasets: Shopee user search click logs
Benchmarks: Shopee Indonesia offline test set (3M items, 9,000 query images)
Metrics: Recall@1 Recall@5 Recall@10 Recall@50 Recall@100 Category Accuracy Same & Similar Item Recall@4 Top 4 Irrelevant Rate Clicks/User Orders/User
Baselines: I2I (ViT-base image embedding) I2T (CLIP-based)
Builds on: CLIP Swin Transformer METER Cross Batch Memory CosFace
Closest papers by meaning
- TIGER-FG: Text-Guided Implicit Fine-Grained Grounding for E-commerce Retrieval (NeurIPS 2026 · similarity 0.87)
- Mixture of Attribute-Aware Attention Experts for Fine-grained E-Commerce Composed Image Retrieval (NeurIPS 2026 · similarity 0.87)
- ICAR: Image-Based Complementary Auto Reasoning (AAAI 2024 · similarity 0.82)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (ICML 2024 · similarity 0.82)
- Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval (ICLR 2025 · similarity 0.81)
- Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.