Research map / Image generation / Text-to-image diffusion
TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TIGeR-ONE, a training-free framework that unifies text-to-image generation and retrieval within one LMM (SEED-LLaMA or LaVIT) using likelihood-based proxies, forward beam search over a visual-token trie, reverse re-ranking, and autonomous decision-making, plus a new TIGeR-Bench.
Problem
Text-to-image retrieval lacks creativity and scales poorly, while text-to-image generation struggles with knowledge-intensive concepts; existing unified approaches rely on external dense retrievers and extra training.
Approach
Uses an LMM's own likelihoods (debiased forward and reverse image-text likelihoods) as similarity proxies, performs retrieval as constrained autoregressive decoding over a trie of discrete image tokens (forward beam search) followed by reverse re-ranking, and runs generation in parallel to choose between generated and retrieved images by the same likelihood scores, all training-free.
Key results
- On TIGeR-Bench unified generation and retrieval, Ours (LaVIT) reaches CLIP-T 28.45 and CLIP-I 61.37, versus 27.07/48.75 for vanilla LaVIT generation.
- On MS-COCO 5K text-to-image retrieval, Ours (SEED-LLaMA) reaches R@1 46.11 vs 37.80 for CLIP ViT-B/32, and training-free.
- On Flickr30K 1K, Ours (SEED-LLaMA) reaches R@1 71.70 vs 68.70 for CLIP ViT-B/32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add generative retrieval and autonomous selection to LaVIT | LaVIT · TIGeR-Bench | CLIP-I: 48.75 -> 61.37 | Helped |
| Use SEED-LLaMA generative retrieval | SEED-LLaMA · MS-COCO (5K) | R@1: 37.80 -> 46.11 | Helped |
| Add retrieval and autonomous selection | SEED-LLaMA · VisDial, 10 rounds | CLIP-I: +3.0 | Helped |
| Add reverse re-ranking (ablation) | TIGeR-ONE with SEED-LLaMA · TIGeR-Bench | R@1: 26.80 -> 29.23 | Helped |
| Add reverse re-ranking (ablation) | TIGeR-ONE with LaVIT · TIGeR-Bench | R@1: 28.13 -> 49.37 | Helped |
| Use reverse rather than forward decision scoring (ablation) | TIGeR-ONE with LaVIT and RRR · TIGeR-Bench | CLIP-I: 49.62 -> 61.37 | Helped |
| Use reverse rather than forward decision scoring (ablation) | TIGeR-ONE with SEED-LLaMA and RRR · TIGeR-Bench | CLIP-T: 23.72 -> 22.84 | Hurt |
| Debias forward likelihood with the unconditional visual prior (ablation) | SEED-LLaMA · MS-COCO | R@1: 3.50 -> 26.25 | Helped |
| Use reverse conditional likelihood instead of forward likelihood (ablation) | SEED-LLaMA · MS-COCO | R@1: 3.50 -> 49.34 | Helped |
| Expand knowledge prompts with GPT-4o (ablation) | SEED-LLaMA · TIGeR-Bench (Knowledge), long text | R@1: 22.57 -> 25.00 | Helped |
| Expand knowledge prompts with Gemini-Pro (ablation) | SEED-LLaMA · TIGeR-Bench (Knowledge), long text | R@1: 22.57 -> 18.57 | Hurt |
| Expand knowledge prompts with self-expansion (ablation) | SEED-LLaMA · TIGeR-Bench (Knowledge), long text | R@1: 22.57 -> 17.20 | Hurt |
Surprising findings
- Reverse-direction likelihood log p(X|Y) (captioning direction) is far more discriminative than the standard forward conditional likelihood, which gives near-random R@1 for LaVIT (0.02)
- LaVIT tends to prefer generated images even when retrieved ones are correct, yielding only 4.38% retrieval selection in one setting.
Limitations
- Retrieval recall is limited by beam size, so FBS can miss target images; reverse re-ranking is needed to close the gap.
- Decision mechanism shows a generation preference in LaVIT, so retrieved correct images are often overridden.
- Only one generated image and top-1 retrieved image are considered per query.
Experimental setup
Datasets: WHOOPS! Pick-a-Pic v2 Logo-2K+ Visual News Google Landmark v2 Food2K iNaturalist WIT Flickr30K MS-COCO VisDial
Benchmarks: TIGeR-Bench Flickr30K MS-COCO VisDial
Metrics: CLIP-T CLIP-I R@1 R@5 R@10 retrieval percentage
Baselines: GILL SDXL Emu 2 DreamLLM SEED-LLaMA LaVIT CLIP (ViT-B/32) GRACE IRGen
Builds on: GILL generative retrieval SEED-LLaMA LaVIT
Closest papers by meaning
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.90)
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant (CVPR 2025 · similarity 0.89)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.88)
- GenIR: Generative Visual Feedback for Mental Image Retrieval (NeurIPS 2025 · similarity 0.88)
- PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction (CVPR 2025 · similarity 0.88)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.