AI Paper Atlas

Research map / Vision-language and multimodal / CLIP and zero-shot

UniIR: Training and Benchmarking Universal Multimodal Information Retrievers

Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, Wenhu Chen

ECCV 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Language models for zero-shot reasoning · 95 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add instruction tuning to multi-task training (CLIP_SF) (ablation)UniIR CLIP_SF · M-BEIR global poolRecall@5 average: 36.1 -> 48.9Improved
add instruction tuning to multi-task training (BLIP_FF) (ablation)UniIR BLIP_FF · M-BEIR global poolRecall@5 average: 34.6 -> 45.5Improved
add instructions to reduce wrong-modality retrieval (ablation)UniIR CLIP_SF · M-BEIR error analysiswrong-modality error rate: 58.8% -> 2.7%Improved
add instruction tuning on within-modality tasks (tasks 2 and 5) (ablation)UniIR · M-BEIR tasks 2 and 5Recall@5: +2.5 and +1.0 (CLIP_SF)No clear effect
multi-task instruction-tuning instead of dataset-specific fine-tuning (BLIP_FF) (ablation)UniIR BLIP_FF · M-BEIR_localRecall@5 average: 37.1 -> 46.8Improved
multi-task instruction-tuning instead of dataset-specific fine-tuning (CLIP_SF) (ablation)UniIR CLIP_SF · M-BEIR_localRecall@5 average: 49.4 -> 50.6Improved
multi-task training without instructions (CLIP_SF) vs dataset-specific (ablation)Multi-task CLIP_SF · M-BEIR_localRecall@5 average: 49.4 -> 49.4No clear effect
add instruction tuning for held-out dataset generalization (ablation)UniIR · M-BEIR held-out (WebQA, OVEN, CIRR)Recall@5Improved
multi-task training without instructions on held-out tasks (ablation)Multi-task baselines · M-BEIR_local held-out WebQA and OVENRecall@5Hurt
use BLIP feature-level fusion instead of score-level fusion for BLIP (ablation)UniIR BLIP · M-BEIR global poolRecall@5 average: 42.7 -> 45.5Improved
use score-level fusion instead of feature-level fusion for CLIP (ablation)UniIR CLIP · M-BEIR global poolRecall@5 average: 43.3 -> 48.9Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy