Research map / Vision-language and multimodal / Human and embodied modeling

Visual Abstraction: A Plug-and-Play Approach for Text-Visual Retrieval

ICML 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Composed and text-based image retrieval

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VISA, a test-time plug-and-play method that converts top-k retrieved images/videos into textual descriptions with off-the-shelf LMMs, refines them via LLM-generated questions and LMM answers, and reranks by text retrieval combined with the original VLM score.

Problem

Text-to-visual retrieval suffers from semantic redundancy in visual features and granularity mismatch between user queries and visual content, and existing fixes require costly training of VLMs or LMMs.

Approach

Abstract visual content into dense natural-language descriptions at test time, then perform query-text similarity in a unified text space; a chain-of-thought question-answering step tailors descriptions to the query's granularity. The method reranks top-k candidates from any existing VLM without training.

Key results

• On MSR-VTT, InternVideo2-G R@1 rises from 52.0 to 54.4 with VISA; on DiDeMo InternVideo2-C R@1 rises from 45.9 to 54.8.
• On DCI, LoTLIP R@1 improves from 62.5 to 74.6 with VISA, and on Urban1k from 85.9 to 94.6.
• On COCO, EVA-CLIP R@1 improves from 55.6 to 59.5 with VISA; on Flickr30K, from 83.1 to 86.1.

Measured effects reported in the paper

Surprising findings

• Combining SigLIP and EVA-CLIP underperforms the VISA-enhanced single model, suggesting larger VLMs do not inherently guarantee better retrieval.
• Qwen2.5-32B-based answering with a 7B answer generator (Qwen2-VL-7B) performs comparably to the 34B LLaVA answer generator.

Limitations

• Relies on off-the-shelf LMM-generated descriptions, which may contain hallucinations and biases (noted in impact statement).
• Offline description generation is costly (about 138K GFLOPs and 437.5 s per dataset for Flickr30K with LLaVA-34B), and online QA adds around 1 s per query.
• Only reranks top-k (k=20) candidates from an existing VLM, so recall is bounded by the base retriever.

Experimental setup

Datasets: COCO Flickr30K MSR-VTT LSMDC DiDeMo MSVD DCI IIW Urban1k ShareGPT4v

Benchmarks: COCO Flickr30K MSR-VTT LSMDC DiDeMo MSVD DCI IIW Urban1k ShareGPT4v

Metrics: R@1 R@5 R@10

Baselines: CoCa FLAME DreamLIP FLAIR UMG-CLIP RAGVL InternVL-G BLIP-2 SigLIP EVA-CLIP CLIP4Clip InternVideo

Builds on: CLIP chain-of-thought prompting LLaVA Qwen2-VL

Closest papers by meaning

• Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models (NeurIPS 2026 · similarity 0.88)
• TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.87)
• Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.86)
• EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models (AAAI 2025 · similarity 0.86)
• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.86)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.86)