AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement

GuangHao Meng, Jinpeng Wang, Qian-Wei Wang, XuDong Ren, Dan Zhao

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a dual-stream multimodal query refinement approach for vision-language retrieval that imagines layouts and sketches to better capture structural and fine-grained entity features of queries.

Problem

Traditional vision-language retrieval relies on shallow data-driven semantic alignment and fails to capture deeper structural and fine-grained entity features, performing poorly on multi-entity layouts and challenging entities.

Approach

Refines queries through a dual-stream multimodal process that generates layout and sketch representations to expose structural and fine-grained entity information before retrieval (based on the abstract only).

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add layout knowledge (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2TR@1: 73.3 -> 75.0Improved
add sketch knowledge (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2TR@1: 73.3 -> 74.7Improved
add both layout and sketch (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2TR@1: 73.3 -> 75.7Improved
add sample-aware gated network (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2TR@1: 75.7 -> 76.3Improved
add progressive contrastive loss (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2TR@1: 76.3 -> 76.6Improved
add progressive contrastive loss (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ T2IR@1: 54.2 -> 54.5Improved
increase number of layout/sketch instances beyond 4 (ablation)LASE (CLIP ViT-B/32) · Flickr30K-CFQ I2TR@1No clear effect
use LASE instead of RACLIPLASE (CLIP ViT-B/32) · N24News I2TR@1: 53.6 -> 54.5Improved
use LASE on unseen domainLASE (CLIP ViT-B/32) · WikiDO OOD I2TR@1: 73.4 -> 74.1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy