Research map / Vision-language and multimodal / CLIP and zero-shot
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, Wenhu Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add instruction tuning to multi-task training (CLIP_SF) (ablation) | UniIR CLIP_SF · M-BEIR global pool | Recall@5 average: 36.1 -> 48.9 | Improved |
| add instruction tuning to multi-task training (BLIP_FF) (ablation) | UniIR BLIP_FF · M-BEIR global pool | Recall@5 average: 34.6 -> 45.5 | Improved |
| add instructions to reduce wrong-modality retrieval (ablation) | UniIR CLIP_SF · M-BEIR error analysis | wrong-modality error rate: 58.8% -> 2.7% | Improved |
| add instruction tuning on within-modality tasks (tasks 2 and 5) (ablation) | UniIR · M-BEIR tasks 2 and 5 | Recall@5: +2.5 and +1.0 (CLIP_SF) | No clear effect |
| multi-task instruction-tuning instead of dataset-specific fine-tuning (BLIP_FF) (ablation) | UniIR BLIP_FF · M-BEIR_local | Recall@5 average: 37.1 -> 46.8 | Improved |
| multi-task instruction-tuning instead of dataset-specific fine-tuning (CLIP_SF) (ablation) | UniIR CLIP_SF · M-BEIR_local | Recall@5 average: 49.4 -> 50.6 | Improved |
| multi-task training without instructions (CLIP_SF) vs dataset-specific (ablation) | Multi-task CLIP_SF · M-BEIR_local | Recall@5 average: 49.4 -> 49.4 | No clear effect |
| add instruction tuning for held-out dataset generalization (ablation) | UniIR · M-BEIR held-out (WebQA, OVEN, CIRR) | Recall@5 | Improved |
| multi-task training without instructions on held-out tasks (ablation) | Multi-task baselines · M-BEIR_local held-out WebQA and OVEN | Recall@5 | Hurt |
| use BLIP feature-level fusion instead of score-level fusion for BLIP (ablation) | UniIR BLIP · M-BEIR global pool | Recall@5 average: 42.7 -> 45.5 | Improved |
| use score-level fusion instead of feature-level fusion for CLIP (ablation) | UniIR CLIP · M-BEIR global pool | Recall@5 average: 43.3 -> 48.9 | Improved |
Closest papers by meaning
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.89)
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.89)
- U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs (ICLR 2026 · similarity 0.89)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.88)
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
