Research map / Vision-language and multimodal / Cross-modal learning
GENIUS: A Generative Framework for Universal Multimodal Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GENIUS, a universal generative retrieval framework that maps multimodal queries plus instructions to discrete modality-decoupled semantic IDs, decoded by a T5-small decoder with query interpolation augmentation and embedding re-ranking.
Problem
Existing generative retrieval methods are task- and modality-specific and underperform embedding-based retrieval, especially in cross-modal settings; a universal multimodal generative retriever is needed.
Approach
Residual quantization over contrastive instruction-aware embeddings where the first code encodes target modality and later codes encode semantics, combined with a T5 decoder conditioned on query prefix embeddings and query augmentation by interpolating query and target embeddings; inference uses trie-constrained beam search followed by embedding re-ranking.
Key results
- On COCO text-to-image retrieval within M-BEIR task pool, GENIUS reaches 68.1 R@5 versus 39.5 for GRACE, and 78.0 with re-ranking.
- In universal retrieval over the 5.6M pool, GENIUS averages 28.8 R@5 vs 48.9 for CLIP-SF; with re-ranking it reaches 38.3.
- On Flickr30K text-to-image, GENIUS-R reaches 92.0 R@5 and COCO 74.0 R@5, outperforming prior generative methods GRACE and IRGen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add modality-decoupled quantization (ablation) | GENIUS · COCO text→image (universal pool) | Recall@5: 20.2 -> 55.4 | Helped |
| Add query augmentation (ablation) | GENIUS · COCO text→image (universal pool) | Recall@5: 47.8 -> 55.4 | Helped |
| Add contrastive loss Lcl (ablation) | GENIUS quantizer · COCO text→image (universal pool) | Recall@5: 0.0 -> 55.4 | Helped |
| Add MSE loss Lmse (ablation) | GENIUS quantizer · COCO text→image (universal pool) | Recall@5: 45.5 -> 55.4 | Helped |
| Increase codebook levels from 5 to 9 (ablation) | GENIUS · WebQA T→T | Recall@5: 17.3 -> 28.8 | Helped |
| Increase codebook size from 256 to 4096 (ablation) | GENIUS · WebQA T→T | Recall@5: 18.3 -> 28.8 | Helped |
| Increase codebook size from 256 to 4096 (ablation) | GENIUS · CIRR (I,T)→I | Recall@5: 18.3 -> 21.0 | Helped |
| Increase codebook levels from 5 to 9 (ablation) | GENIUS · CIRR (I,T)→I | Recall@5: 20.4 -> 21.0 | Helped |
| Use a larger codebook and more levels (ablation) | GENIUS · WebQA T→(I,T) | Recall@5: 22.8 -> 47.4 | Helped |
Surprising findings
- Generative retrieval remains well behind embedding-based baselines on knowledge-intensive tasks such as WebQA and InfoSeek.
- Removing the contrastive loss drops performance to near zero across all tasks.
Limitations
- Underperforms embedding-based retrieval on knowledge-intensive tasks (WebQA, InfoSeek).
- Gap to embedding-based methods remains in universal retrieval without re-ranking.
- Performance depends on codebook size and level configuration, which needs dataset-specific balancing.
Experimental setup
Datasets: M-BEIR MS-COCO Flickr30K Fashion200K FashionIQ VisualNews NIGHTS OVEN EDIS CIRR InfoSeek WebQA
Benchmarks: M-BEIR Flickr30K MS-COCO
Metrics: Recall@5 Recall@10 Recall@1 queries per second
Baselines: CLIP-SF BLIP-FF GRACE IRGen
Builds on: DSI GRACE IRGen UniIR CLIP
Closest papers by meaning
- Closing the Indexing-Decoding Gap in Multimodal Generative Retrieval via Prefix Retention Optimization (NeurIPS 2026 · similarity 0.88)
- MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.87)
- Generative Retrieval Meets Multi-Graded Relevance (NeurIPS 2024 · similarity 0.87)
- Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- GenIR: Generative Visual Feedback for Mental Image Retrieval (NeurIPS 2025 · similarity 0.86)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.