Research map / Image generation / Autoregressive and latent generation
DiffusionREC: Diffusion Model with Adaptive Condition for Referring Expression Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies a DDPM-style text-conditioned bounding-box diffusion model with a filtering-based object decoder to referring expression comprehension, iteratively denoising and pruning randomly sampled boxes.
Problem
Referring expression comprehension (REC) methods rely on learnable queries (hard to converge, weak on small objects) or detector proposals (failure when the detector misses the target); diffusion has not been applied to REC.
Approach
Treat REC as text-guided bounding-box denoising: noisy boxes are sampled at inference and refined over steps by a filtering-based object decoder conditioned on the expression, previous-step box features, and vision-language features; a vision-guided text prompting module selects key text tokens and a vision-language filtering module prunes irrelevant boxes.
Key results
- DiffusionREC reaches 89.35% on RefCOCO val and 87.28% on RefCOCO testB (per Table 1 and 2), with COCO pretraining.
- DiffusionREC improves about 1.46% over OFA-base on RefCOCOg test, per the paper.
- Replacing DDPM with DDIM (50 steps) gives similar accuracy with much faster inference (1.13s vs 22.6s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffusionREC instead of CyCo | REC models · RefCOCO val | accuracy: 89.47 -> 89.35 | Hurt |
| Use DiffusionREC instead of CyCo | REC models · RefCOCO testA | accuracy: 91.87 -> 91.54 | Hurt |
| Use DiffusionREC instead of CyCo | REC models · RefCOCO testB | accuracy: 85.33 -> 87.28 | Helped |
| Use DiffusionREC instead of MDETR | REC models · RefCOCO+ val | accuracy: 79.52 -> 83.04 | Helped |
| Use DiffusionREC instead of MDETR | REC models · RefCOCO+ testA | accuracy: 84.09 -> 87.81 | Helped |
| Use DiffusionREC instead of OFA-base | REC models · RefCOCO+ testB | accuracy: 74.29 -> 72.79 | Hurt |
| Use DiffusionREC instead of LADS | REC models · RefCOCOg val | accuracy: 82.67 -> 83.94 | Helped |
| Use DiffusionREC instead of LADS | REC models · RefCOCOg test | accuracy: 81.96 -> 83.50 | Helped |
| Use DiffusionREC instead of MDETR | REC models · Flickr30K test | accuracy: 83.80 -> 86.67 | Helped |
| Use DiffusionREC instead of LADS | REC models · RefClef test | accuracy: 78.82 -> 78.14 | Hurt |
| Add the vision-language filtering module (ablation) | DiffusionREC · RefCOCO testA | accuracy: 87.68 -> 91.54 | Helped |
| Add the vision-language filtering module (ablation) | DiffusionREC · RefCOCO testB | accuracy: 84.02 -> 87.28 | Helped |
Surprising findings
- Removing the vision-language filtering module drops RefCOCO testA from 91.54 to 87.68.
- Using a single full-image box instead of 1,000 random boxes lowers average accuracy by about 6.6 points.
Limitations
- Performance is lower than some baselines on RefCOCO+ testB (e.g., OFA-base and others).
- Failure cases occur when target objects are similar to neighboring objects.
- Pre-training on COCO is used for the reported best results, and inference still requires many denoising steps.
Experimental setup
Datasets: RefCOCO RefCOCO+ RefCOCOg Flickr30K Entities RefClef
Benchmarks: RefCOCO RefCOCO+ RefCOCOg Flickr30K Entities RefClef
Metrics: accuracy at IoU>0.5
Baselines: M-DGT QRNet TransVG++ MDETR OFA-base VG-LAW SMCIM LGR-NET Refcrowd CLIPREC CyCo LADS
Builds on: DiffusionDet DDPM Sparse R-CNN DDIM
Closest papers by meaning
- ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.89)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.87)
- Revisiting Counterfactual Problems in Referring Expression Comprehension (CVPR 2024 · similarity 0.87)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.87)
- Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.