Research map / Vision-language and multimodal / Multimodal LLMs
Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement
GuangHao Meng, Jinpeng Wang, Qian-Wei Wang, XuDong Ren, Dan Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a dual-stream multimodal query refinement approach for vision-language retrieval that imagines layouts and sketches to better capture structural and fine-grained entity features of queries.
Problem
Traditional vision-language retrieval relies on shallow data-driven semantic alignment and fails to capture deeper structural and fine-grained entity features, performing poorly on multi-entity layouts and challenging entities.
Approach
Refines queries through a dual-stream multimodal process that generates layout and sketch representations to expose structural and fine-grained entity information before retrieval (based on the abstract only).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add layout knowledge (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2T | R@1: 73.3 -> 75.0 | Improved |
| add sketch knowledge (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2T | R@1: 73.3 -> 74.7 | Improved |
| add both layout and sketch (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2T | R@1: 73.3 -> 75.7 | Improved |
| add sample-aware gated network (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2T | R@1: 75.7 -> 76.3 | Improved |
| add progressive contrastive loss (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2T | R@1: 76.3 -> 76.6 | Improved |
| add progressive contrastive loss (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ T2I | R@1: 54.2 -> 54.5 | Improved |
| increase number of layout/sketch instances beyond 4 (ablation) | LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2T | R@1 | No clear effect |
| use LASE instead of RACLIP | LASE (CLIP ViT-B/32) · N24News I2T | R@1: 53.6 -> 54.5 | Improved |
| use LASE on unseen domain | LASE (CLIP ViT-B/32) · WikiDO OOD I2T | R@1: 73.4 -> 74.1 | Improved |
Closest papers by meaning
- EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Models (AAAI 2025 · similarity 0.89)
- Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval (NeurIPS 2025 · similarity 0.88)
- Generative Multi-Modal Knowledge Retrieval with Large Language Models (AAAI 2024 · similarity 0.87)
- O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model (AAAI 2026 · similarity 0.87)
- Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
