Research map / Vision-language and multimodal / Human and embodied modeling
MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds 36.7M (query image, open-ended instruction, target image) triplets by mining same-web-page image pairs from Common Crawl and generating instructions with LMM+LLM metadata and PaLM2, then trains light dual-encoder MagicLens models (CLIP/CoCa initialized) with contrastive loss for instruction-following image retrieval.
Problem
Image retrieval given a reference image and an open-ended text instruction is limited because existing methods rely on visually similar pairs, pre-defined relations, or template-based synthetic instructions, failing to capture rich non-visual search intents.
Approach
Use naturally co-occurring image pairs from the same web page as self-supervised positives, and convert their implicit relations into free-form instructions with LMM metadata and LLM generation, then train a dual-encoder with shared parameters and self-attention fusion, including query images as hard negatives.
Key results
- MagicLens-L (CoCa-L, 613M) reaches 34.1 mAP@5 on CIRCO versus 12.6 for prior CLIP-L SOTA (LinCIR), and 48.2 R@10 on DTIN versus 12.9 for Context-I2W.
- MagicLens-B (CoCa-B, 267M) outperforms CIReVL (12.3B+ parameters) on CIRCO, DTIN and GeneCIS, with a roughly 50x smaller model.
- In a human one-on-one evaluation on a 1.4M-image pool, MagicLens-L wins 80.0% vs 4.7% on beyond-visual instructions and 61.3% vs 24.0% on complex visual instructions against LinCIR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on naturally mined data instead of IP2P data at 1M scale (ablation) | CoCa-based MagicLens-B · FIQ, CIRR, CIRCO, DTIN, GeneCIS | benchmark score (R@10, R@1, mAP@5, R@10, R@1): 20.3 -> 33.5; 12.5 -> 29.6; 13.6 -> 29.7; 30.2 -> 43.7; 14.5 -> 15.8 | Helped |
| Use template-free rather than template-based instructions (ablation) | CoCa-based MagicLens-B at 1M scale · FIQ, CIRR, CIRCO, DTIN, GeneCIS | benchmark score (R@10, R@1, mAP@5, R@10, R@1): 33.4 -> 33.5; 23.4 -> 29.6; 25.1 -> 29.7; 23.1 -> 43.7; 14.6 -> 15.8 | Helped |
| Add query-image hard negatives during training (ablation) | MagicLens-B · FIQ, CIRR, CIRCO, DTIN, GeneCIS | benchmark score (R@10, R@1, mAP@5, R@10, R@1): 33.2 -> 35.2; 1.6 -> 31.6; 11.9 -> 30.8; 14.1 -> 46.8; 14.5 -> 17.4 | Helped |
| Replace self-attention with cross-attention (ablation) | MagicLens-B and MagicLens-L · FIQ, CIRR, CIRCO, DTIN, GeneCIS | benchmark score: MagicLens-B: 35.2 -> 31.0, 31.6 -> 28.3, 30.8 -> 27.0, 46.8 -> 41.4, 17.4 -> 16.2; MagicLens-L: 38.0 -> 32.3, 33.3 -> 29 | Mixed |
| Freeze the backbone vision and language encoders during training (ablation) | MagicLens-B and MagicLens-L · FIQ, CIRR, CIRCO, DTIN, GeneCIS | benchmark score: MagicLens-B: 35.2 -> 30.8, 31.6 -> 25.9, 30.8 -> 21.7, 46.8 -> 30.1, 17.4 -> 15.2; MagicLens-L: 38.0 -> 32.5, 33.3 -> 26 | Hurt |
| Increase training data size (ablation) | CoCa-based MagicLens-B · Five multimodality-to-image benchmarks (average performance) | average performance | Helped |
| Scale from MagicLens-B to MagicLens-L | MagicLens · FIQ, CIRR, CIRCO, DTIN, GeneCIS | benchmark score: 35.2 -> 38.0; 31.6 -> 33.3; 30.8 -> 34.1; 46.8 -> 48.2; 17.4 -> 16.7 | Mixed |
| Train MagicLens on the constructed triplets | CoCa-B backbone · Flickr30K and MSCOCO text-to-image retrieval | R@1, R@5, R@10: Flickr30K: 74.5 -> 76.2, 93.4 -> 93.7, 96.4 -> 96.5; MSCOCO: 46.4 -> 48.9, 71.5 -> 73.9, 80.1 -> 82.5 | Helped |
| Train MagicLens on the constructed triplets | CoCa-L backbone · Flickr30K and MSCOCO image-to-text retrieval | R@1, R@5, R@10: Flickr30K: 92.1 -> 89.6, 98.8 -> 98.7, 99.9 -> 99.4; MSCOCO: 65.1 -> 67.7, 85.5 -> 87.6, 91.3 -> 92.7 | Mixed |
| Use MagicLens-L rather than LinCIR | 1.4M-image holdout retrieval · Simple Visual instructions | human win rate: 41.3 -> 50.7 | Helped |
| Use MagicLens-L rather than LinCIR | 1.4M-image holdout retrieval · Complex Visual instructions | human win rate: 24.0 -> 61.3 | Helped |
| Use MagicLens-L rather than LinCIR | 1.4M-image holdout retrieval · Beyond Visual instructions | human win rate: 4.7 -> 80.0 | Helped |
Surprising findings
- Removing query images as hard negatives collapses CIRR R@1 from 31.6 to 1.6 and CIRCO mAP@5 from 30.8 to 11.9, with the model preferring the query image itself.
- Training on the authors' data also improves text-to-image retrieval of the backbone encoders on Flickr30K and MSCOCO, while image-to-text recall shows marginal drops.
Limitations
- Human evaluation uses only 150 manually written queries (50 per instruction type) with three evaluators on a single held-out pool.
- Relies on proprietary components (PaLI, PaLM2, Google Vision and Natural Language APIs) for data construction, limiting reproducibility.
- Model may return qualified but imperfect results when instructions are under-specified, as shown in the 3D anatomy example.
Experimental setup
Datasets: Common Crawl web pages (36.7M mined triplets) InstructPix2Pix data (1M, ablation) CIRR CIRCO FIQ DTIN GeneCIS Flickr30K MSCOCO TU-Berlin Sketchy QuickDraw
Benchmarks: FIQ CIRR CIRCO DTIN GeneCIS TU-Berlin Sketchy QuickDraw Flickr30K MSCOCO
Metrics: Recall@K mAP@5 win rate precision@K
Baselines: PALAVRA Pic2Word SEARLE Context-I2W LinCIR CIReVL CompoDiff PLI
Builds on: CoCa CLIP PaLI InstructPix2Pix
Closest papers by meaning
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.89)
- INQUIRE: A Natural World Text-to-Image Retrieval Benchmark (NeurIPS 2024 · similarity 0.87)
- ILIAS: Instance-Level Image retrieval At Scale (CVPR 2025 · similarity 0.87)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.87)
- Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark (CVPR 2025 · similarity 0.86)
- Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.