Research map / Vision-language and multimodal / Human and embodied modeling
MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MERIT, a multilingual interleaved multi-condition semantic retrieval benchmark for product search, and CoMRAL, a fine-tuning framework that adds masked embedding reconstruction to contrastive learning for MLLM-based retrievers.
Problem
Existing semantic retrieval datasets are single-language, single-image, or single-condition, and MLLM-based retrievers trained with EOS-token contrastive learning neglect fine-grained conditional query elements and fail to use interleaved image inputs.
Approach
Combine InfoNCE contrastive learning on the EOS embedding with masked multimodal embedding reconstruction (vision and language decoders) so the retriever preserves fine-grained conditional details; build a 5-language, 320K-pair benchmark with interleaved multi-image, multi-condition queries.
Key results
- Best existing retriever on MERIT reaches only 12.05% R@1 (LamRA-Qwen2.5VL 7B, concatenated input).
- CoMRAL (contrastive plus reconstruction) on Qwen2.5-VL with full fine-tuning reaches 69.68% R@1 on MERIT sequential input, a 45.9% improvement over contrastive learning alone.
- Sequential image input improves 14.3% after training on MERIT, supporting the hypothesis that prior single-image retrieval datasets degrade interleaved input handling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multimodal embedding reconstruction to contrastive learning | Qwen2.5-VL retrieval fine-tuning on MERIT · MERIT | R@1: 47.76 -> 69.68 | Helped |
| Use full-parameter rather than LoRA fine-tuning (ablation) | MERIT · MERIT | R@1: 59.40 -> 69.68 | Helped |
| Use sequential rather than concatenated image input (ablation) | MERIT · MERIT | R@1: 60.94 -> 69.68 | Helped |
| Add vision reconstruction to contrastive learning (ablation) | Qwen2.5-VL with LoRA on MERIT · MERIT | R@1: 48.52 -> 58.18 | Helped |
| Add language reconstruction to contrastive learning (ablation) | Qwen2.5-VL with LoRA on MERIT · MERIT | R@1: 48.52 -> 58.38 | Helped |
| Add vision reconstruction to contrastive learning (ablation) | Qwen2.5-VL with full fine-tuning on MERIT · MERIT | R@1: 47.76 -> 59.46 | Helped |
| Add language reconstruction to contrastive learning (ablation) | Qwen2.5-VL with full fine-tuning on MERIT · MERIT | R@1: 47.76 -> 59.98 | Helped |
| Reconstruct both modalities rather than only vision or language (ablation) | Qwen2.5-VL retrieval fine-tuning on MERIT · MERIT | R@1: 59.98 -> 69.68 | Helped |
| Train with sequential image input instead of concatenated images | GME-Qwen2VL · MERIT | R@5: 119.7% improvement | Hurt |
| Train on MERIT to support sequential image inputs | MLLM-based retrieval models · MERIT | R@1: 14.3% | Helped |
| Replace images with corresponding captions | BGE-VL · MERIT | retrieval performance | Hurt |
| Remove product images | BGE-VL · MERIT | retrieval performance: 73.9% | Hurt |
Surprising findings
- Replacing images with captions barely hurts performance on FashionIQ and CIRR, indicating under-use of visual information in existing benchmarks.
- Existing MLLM retrievers perform roughly 16x better in R@1 with concatenated images than with interleaved image inputs.
Limitations
- Benchmark is built from a proprietary internal product pool across Southeast Asian markets and focuses on product retrieval.
- Data pipeline relies heavily on LLM (GPT-4o) annotation and filtering with substantial manual effort.
Experimental setup
Datasets: MERIT
Benchmarks: MERIT CIRR FashionIQ VisDial 8 established retrieval benchmarks
Metrics: R@1 R@5 R@10 MRR
Baselines: InternVL2.5 InternVL2.5-MPO Qwen2.5-VL E5-V LLaVE GME-Qwen2VL LamRA-Qwen2.5VL BGE-VL VLM2Vec
Builds on: InfoNCE E5-V LamRA GME
Closest papers by meaning
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.87)
- INQUIRE: A Natural World Text-to-Image Retrieval Benchmark (NeurIPS 2024 · similarity 0.86)
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark (NeurIPS 2024 · similarity 0.86)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.86)
- CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.