Research map / Vision-language and multimodal / CLIP and zero-shot
MOFI: Learning Image Representations from Noisy Entity Annotated Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the I2E dataset (about 1.1B images, 2M entities) by NER/entity-linking alt-text and CLIP-filtering entities, then trains MOFI with a multi-task supervised entity classification plus contrastive objective for image representations.
Problem
Learning general-purpose image representations at scale without expensive human labels; existing web image-text training (CLIP) cannot use classification objectives and lacks structured fine-grained labels.
Approach
Automatically assign Wikidata entity labels to noisy web image-text pairs using NER/entity linking plus CLIP-based filtering, then combine large-margin supervised classification over ~2M entities with contrastive image-text training, with entity names enriched by descriptions as text.
Key results
- MOFI-H/14 reaches 86.67 mAP@all on GPR1200 and MOFI-L/14 reaches 86.15, versus 72.19 for OpenAI CLIP-L/14.
- MOFI-B/16 achieves 72.99% zero-shot ImageNet accuracy vs 68.38% for OpenAI CLIP-B/16 and 81.32% linear probe vs 80.2% for OpenAI CLIP-B/16.
- Switching CLIP training from I2T to I2E raises GPR1200 mAP from 65.94 to 76.14 for B/16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace I2T training data with I2E for contrastive pre-training | CLIP-B/16 · GPR1200 | mAP@all: 65.94 -> 76.14 | Helped |
| Replace I2T training data with I2E for contrastive pre-training | CLIP-B/16 · ImageNet zero-shot classification | top-1 accuracy: 68.72 -> 72.94 | Helped |
| Replace I2T training data with I2E for contrastive pre-training | CLIP-B/16 · VTAB (9-task average) | zero-shot top-1 accuracy: 64.59 -> 65.41 | Mixed |
| Add supervised entity classification to contrastive pre-training | MOFI-B/16 · GPR1200 | mAP@all: 76.14 -> 83.33 | Helped |
| Add supervised entity classification to contrastive pre-training | MOFI-B/16 · ImageNet linear probe | top-1 accuracy: 79.54 -> 81.32 | Helped |
| Add supervised entity classification to contrastive pre-training | MOFI-B/16 · ImageNet zero-shot classification | top-1 accuracy: 72.94 -> 72.99 | No clear effect |
| Train using combined I2E and I2T data instead of I2E alone | CLIP-B/16 · GPR1200 | mAP@all: 76.14 -> 73.98 | Hurt |
| Train using combined I2E and I2T data instead of I2E alone | MOFI-B/16 · GPR1200 | mAP@all: 83.33 -> 83.41 | No clear effect |
| Use MOFI instead of CLIP-B/16 (I2T) | MOFI-B/16 · ImageNet-1K retrieval | Acc@1: +10.84 | Helped |
| Use MOFI instead of CLIP-B/16 (I2T) | MOFI-B/16 · GPR1200 SOP domain | mAP@all: -4.5 | Hurt |
| Increase supervised-pretraining negative entities up to 32k (ablation) | MOFI supervised pre-training · Image retrieval and zero-shot classification | performance | Helped |
| Increase supervised-pretraining negative entities beyond 32k (ablation) | MOFI supervised pre-training · Image retrieval and zero-shot classification | performance | No clear effect |
Surprising findings
- MOFI underperforms the CLIP model trained on I2T on the SOP product retrieval domain (-4.5 points), attributed to difficulty extracting fine-grained entities from product text.
- MOFI models struggle on specialized VTAB tasks (e.g., Eurosat, Camelyon) in zero-shot evaluation.
Limitations
- Entity extraction from short or product-related text is less precise, hurting SOP retrieval.
- Zero-shot performance lags on specialized VTAB tasks.
- The I2E dataset is not released; the tail entities' benefit beyond 1M entities is not shown.
Experimental setup
Datasets: I2E I2T ImageNet-1K ImageNet-21K GPR1200 VTAB Wikidata GLDv2
Benchmarks: GPR1200 ImageNet-1K VTAB ROxford RParis
Metrics: mAP@all kNN accuracy Recall@1 Recall@5 top-1 accuracy linear probe accuracy
Baselines: CLIP-B/16 OpenAI CLIP-L/14 OpenAI DINOv2 Swin-B ViT-L DOLG MSTFC
Builds on: CLIP ALIGN Large margin cosine loss K-Lite
Closest papers by meaning
- Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach (NeurIPS 2024 · similarity 0.86)
- Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking (AAAI 2025 · similarity 0.85)
- SOHES: Self-supervised Open-world Hierarchical Entity Segmentation (ICLR 2024 · similarity 0.85)
- A Generative Approach for Wikipedia-Scale Visual Entity Recognition (CVPR 2024 · similarity 0.84)
- MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions (ICML 2024 · similarity 0.84)
- ENCODER: Entity Mining and Modification Relation Binding for Composed Image Retrieval (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.