Research map / Vision-language and multimodal / Human and embodied modeling
Seeing Through Words: Controlling Visual Retrieval Quality with Language
Jianglin Lu, Simon Jenni, Kushal Kafle, Jing Shi, Handong Zhao, Yun Fu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| condition LLM query completion on quality levels instead of random scores (ablation) | QCQC · Flickr2.4M, CoCa, Aes High/Rel High | Ave Rel: 0.365 -> 0.390 | Improved |
| complete short queries with fine-tuned quality-conditioned LLM instead of prefix only | QCQC · Flickr2.4M, CoCa, Aes High/Rel High | Ave Aes: 4.735 -> 5.270 | Improved |
| complete short queries with QCQC instead of prefix only | QCQC · MS-COCO, Aes High/Rel High | Ave Aes: 4.817 -> 5.024 | Improved |
| use pretrained LLM for completion without finetuning (ablation) | PT baseline · Flickr2.4M and MS-COCO | Ave Aes / Ave Rel | Hurt |
| fine-tune LLM on image descriptions (random quality conditions) (ablation) | FT baseline · Flickr2.4M, CoCa | Ave Rel: 0.350 -> 0.365 | Improved |
| raise quality condition from Low to High (ablation) | QCQC · Flickr2.4M, CoCa, Rel Low vs Rel High | Ave Rel: 0.355 -> 0.391 | Improved |
| raise aesthetic condition from Low to High (ablation) | QCQC · Flickr2.4M, CoCa, Aes L vs H | Ave Aes: 4.458 -> 5.222 | Improved |
| use QCQC instead of GPT-4o completion | QCQC · Flickr2.4M, CoCa, Aes High/Rel High | Ave Rel: 0.361 -> 0.390 | Improved |
| use QCQC instead of GPT-4o completion on MS-COCO | QCQC · MS-COCO, Aes High/Rel High | Ave Aes: 5.048 -> 5.024 | No clear effect |
| use QCQC instead of post-retrieval aesthetic reranking | QCQC · Flickr2.4M, k=10 | Ave Rel: 0.341 -> 0.366 | Improved |
| increase rerank candidate pool k from 1 to 10 (ablation) | Rerank baseline · Flickr2.4M | Ave Aes / Ave Rel: Aes 4.735 -> 5.313, Rel 0.350 -> 0.341 | Mixed |
| use five quality levels instead of three (ablation) | QCQC · Flickr2.4M, CoCa | Ave Aes | Improved |
Closest papers by meaning
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.90)
- Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms (NeurIPS 2024 · similarity 0.89)
- EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models (AAAI 2025 · similarity 0.89)
- BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval (NeurIPS 2024 · similarity 0.89)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.88)
- The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
