Research map / Image generation / Text-to-image diffusion
RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RealRAG, a retrieval-augmented text-to-image framework whose retriever is trained with self-reflective contrastive learning (generator-generated images used as hard negatives) to fetch real-object images that fill the generator's knowledge gaps.
Problem
Text-to-image generators store visual knowledge in fixed parameters trained on closed datasets, so they hallucinate or distort fine-grained and newly appearing real-world objects; similarity-based retrieval does not supply the missing knowledge.
Approach
Train a reflective retriever on top of CLIP by treating the generator's own outputs as hard negatives (the real-object images most similar to generated images), so retrieved references lie outside the generator's generation space yet near the text prompt; the retrieved image is then fed to the frozen generator (via ControlNet for diffusion models, directly for AR models).
Key results
- Averaged over generators, RealRAG improves Stanford Cars FID by about 6.19% and reaches a 16.18% FID gain with the AR model Emu on Stanford Cars (per abstract).
- On Stanford Cars with Emu, FID drops from 86.73 to 70.55 and CLIP-T rises from 13.49 to 15.30.
- In a 26-participant human evaluation on novel objects, RealRAG images received mean Likert scores above 5 for all participants, versus Flux baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RealRAG to SD V2.1 | SD V2.1 · Stanford Cars | CLIP-I: 61.48 -> 64.50 | Helped |
| Apply RealRAG to Flux | Flux · Stanford Cars | FID: 58.47 -> 52.28 | Helped |
| Apply RealRAG to SD V2.1 | SD V2.1 · Stanford Dogs | CLIP-I: 68.80 -> 79.06 | Helped |
| Apply RealRAG to SD V2.1 | SD V2.1 · Oxford Flowers | CLIP-I: 82.97 -> 88.43 | Helped |
| Replace standard contrastive learning with self-reflective contrastive learning (ablation) | Emu RealRAG · Stanford Cars | FID: 76.85 -> 70.55 | Helped |
| Replace standard contrastive learning with self-reflective contrastive learning (ablation) | Emu RealRAG · Stanford Cars | CLIP-T: 14.31 -> 15.30 | Helped |
| Train the retriever with standard contrastive learning instead of using pre-trained CLIP retrieval (ablation) | Emu RAG · Stanford Cars | FID: 80.75 -> 76.85 | Helped |
| Replace standard contrastive learning with self-reflective contrastive learning (ablation) | Emu RAG · Three fine-grained datasets | Average accuracy: 38.98 -> 39.90 | Helped |
| Replace standard contrastive learning with self-reflective contrastive learning (ablation) | SD V2.1 RAG · Three fine-grained datasets | Average accuracy: 39.73 -> 40.23 | Helped |
| Replace standard contrastive learning with self-reflective contrastive learning (ablation) | Flux RAG · Three fine-grained datasets | Average accuracy: 39.27 -> 40.37 | Helped |
Surprising findings
- Retrieving the image with highest text similarity does not help generation; the most useful references are ones with lower similarity that supply missing knowledge.
- RealRAG converges more slowly than similarity-based retrieval in early training epochs, only surpassing the generative bottleneck in later epochs.
Limitations
- Evaluated mainly on three fine-grained datasets (cars, dogs, flowers) with a fixed retrieval database of their training sets.
- Diffusion models require ControlNet branches to accept retrieved images as conditions, adding architectural complexity.
- Human evaluation involves only 26 participants and a small number of novel-object cases.
Experimental setup
Datasets: ImageNet Stanford Cars Stanford Dogs Oxford Flowers Google Images (novel objects)
Benchmarks: Stanford Cars Stanford Dogs Oxford Flowers
Metrics: FID CLIP-I CLIP-T classification accuracy (OpenCLIP) 7-point Likert human score
Baselines: OmniGen Emu Stable Diffusion V2.1 SDXL Stable Diffusion V3 Flux Zero-RAG Normal-RAG
Builds on: CLIP ControlNet contrastive learning for retrieval RDM (retrieval-augmented diffusion models)
Closest papers by meaning
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.88)
- The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better (NeurIPS 2024 · similarity 0.87)
- RobustVisRAG: Causality-Aware Vision-Based Retrieval-Augmented Generation under Visual Degradations (CVPR 2026 · similarity 0.87)
- AR-RAG: Autoregressive Retrieval Augmentation for Image Generation (NeurIPS 2025 · similarity 0.87)
- Compositional Scene Understanding through Inverse Generative Modeling (ICML 2025 · similarity 0.86)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.