Research map / Vision-language and multimodal / Human and embodied modeling
GenIR: Generative Visual Feedback for Mental Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mental Image Retrieval (MIR) and GenIR, which uses text-to-image diffusion-generated images as visual feedback in multi-round retrieval, plus an automated pipeline to build a multi-round MIR dataset.
Problem
Interactive text-to-image retrieval where a user refines queries toward a mentally held target image; existing verbal feedback methods (ChatIR, PlugIR) give abstract or misleading feedback that is hard for users to act on.
Approach
Generate a synthetic image from the current text query with a diffusion model, show it to the user as explicit visual feedback, and use image-to-image retrieval with the synthetic image as the query, so each round reifies the system's understanding.
Key results
- On MSCOCO (50k search space), GenIR with Infinity reaches roughly 98% Hits@10 by round 10, versus about 92% for Prediction Feedback and Gemma3-12b verbal feedback and about 73% for ChatIR.
- On FFHQ and Clothing-ADC, GenIR reaches 70% and 73% Hits@10 versus 52% and 50% for the next best method.
- In a 100-sample human study, 86% of ninth-round generated images were judged useful for query refinement (single annotator).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace verbal ChatIR text feedback with GenIR synthetic-image feedback | MSCOCO retrieval · MSCOCO, dialog length 10 | Hits@10 (%): 73.64 -> 98.01 | Helped |
| Use GenIR-refined text queries instead of ChatIR verbal-feedback queries | Text-to-image retrieval · MSCOCO, dialog length 10 | Hits@10 (%): 73.64 -> 92.33 | Helped |
| Use generative visual feedback rather than the next best method | GenIR · FFHQ | Hits@10: 52% -> 70% | Helped |
| Use generative visual feedback rather than the next best method | GenIR · Clothing-ADC | Hits@10: 50% -> 73% | Helped |
| Use GenIR visual feedback rather than verbal feedback | MSCOCO retrieval · MSCOCO, round 5 | Hits@10: 89.97% -> 96.85% | Helped |
| Use visual feedback for 22.3% of queries and verbal feedback for 77.7% (random selection) | Hybrid system · MSCOCO, dialog length 10 | Hits@10 (%): 92.33 -> 93.62 | Helped |
| Use oracle selection between visual and verbal feedback | Hybrid system · MSCOCO, dialog length 10 | Hits@10 (%): 92.33 -> 98.88 | Helped |
| Increase the VLM size from Gemma3-4b to Gemma3-12b (ablation) | Feedback methods · MSCOCO and FFHQ | Hits@10 | Helped |
| Use GenIR with Gemma3-4b instead of alternative feedback methods with Gemma3-12b (ablation) | MSCOCO and FFHQ retrieval · MSCOCO and FFHQ | Hits@10 | Helped |
| Replace the verbal feedback baseline with prediction feedback (ablation) | Interactive retrieval · MSCOCO | Hits@10 | Mixed |
| Vary the text-to-image generator within GenIR (ablation) | GenIR · MSCOCO | Hits@10 | No clear effect |
| Use synthetic-image retrieval instead of GenIR-refined text queries (ablation) | GenIR retrieval · MSCOCO, dialog length 10 | Hits@10 (%): 92.33 -> 98.01 | Helped |
Surprising findings
- Prediction feedback initially beats verbal feedback but plateaus after 2-4 rounds and is eventually surpassed by text-only methods in longer dialogues.
- Generator choice matters little: performance differences among diffusion models are small compared with the gap to baselines.
Limitations
- Uses a VLM (Gemma3) to simulate the user with a fixed clear target image; no real human-in-the-loop study at scale.
- Generated images can hallucinate or misrepresent retrieval-critical details, and improvements stall in later rounds.
- Per-round latency is about 8x that of verbal feedback.
Experimental setup
Datasets: MS COCO FFHQ Flickr30k Clothing-ADC
Benchmarks: MS COCO FFHQ Flickr30k Clothing-ADC
Metrics: Hits@10 human usefulness rate
Baselines: ChatIR PlugIR Verbal Feedback (Gemma3) Prediction Feedback
Builds on: ChatIR PlugIR Imagine-and-Seek
Closest papers by meaning
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.88)
- Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG (ICML 2025 · similarity 0.88)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.87)
- PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction (CVPR 2025 · similarity 0.87)
- CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception (ICML 2026 · similarity 0.87)
- GENIUS: A Generative Framework for Universal Multimodal Search (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.