Research map / Vision-language and multimodal / Multimodal LLMs

SLQ: Bridging Modalities via Shared Latent Queries for Retrieval with Frozen MLLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SLQ, which adapts frozen MLLMs (InternVL3, Qwen3-VL) into multimodal retrievers by appending a small set of shared learnable queries and training only those queries with symmetric InfoNCE, and introduces KARR-Bench, a knowledge-aware reasoning retrieval benchmark built from COCO.

Problem

Full fine-tuning and LoRA of generative MLLMs for dense retrieval can distort pre-trained semantic space and cost heavy compute, and standard retrieval benchmarks mostly test surface-level pattern matching rather than knowledge-aware reasoning.

Approach

Freeze the entire MLLM and learn only N=20 queries appended to both image and text inputs; mean-pool their final-layer hidden states via causal attention to form a shared embedding, avoiding external heads or separate modality-specific queries.

Key results

• On COCO 5K text-to-image, SLQ (InternVL3-8B) reaches R@1 55.4 vs. VLM2VEC-7B 49.2 and E5-V-7B 52.0.
• On MMEB overall (36 tasks), SLQ-8B (fine-tuned on MMEB-train) scores 67.5, above UniME-7B (66.6) and VLM2VEC-7B (62.9).
• SLQ trains only 36K parameters on InternVL3-1B, cutting GPU hours from 11.4 (LoRA) to 3.9; on InternVL3-8B, 144K parameters and 38.9 vs 130.1 GPU hours.

Surprising findings

• Full fine-tuning and LoRA degrade VQA performance on MMMU, RealWorldQA and OCRBench relative to the frozen backbone, which SLQ preserves or improves.
• External projection heads (linear or Transformer block) on the last token perform worse than the untrained last-token baseline pattern, and Prompt Tuning performs worst among PEFT methods.

Limitations

• Backbone is frozen, which may limit adaptation to extreme out-of-distribution domains such as medical or satellite imagery.
• Only validated with symmetric InfoNCE; triplet loss, listwise ranking, and distillation are unexplored.
• KARR-Bench is derived from 5,000 COCO test images with GPT-5-mini generated queries and a 60-70% human acceptance rate, so it is small (2,915 pairs).

Experimental setup

Datasets: COCO Flickr30K MMEB-train MMEB FashionIQ CIRR

Benchmarks: Flickr30K COCO MMEB KARR-Bench MMMU RealWorldQA OCRBench FashionIQ CIRR

Metrics: Recall@1 Recall@5 Recall@10 Precision (MMEB) GPU hours

Baselines: CLIP BLIP FLAME E5-V VLM2Vec TIGeR LoRA Full fine-tuning

Builds on: LoRA InfoNCE VLM2Vec GME

Closest papers by meaning

• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
• Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval (NeurIPS 2025 · similarity 0.88)
• Retriever-Free Retrieval-Augmented Reasoning via Corpus-Traversing MCTS (NeurIPS 2026 · similarity 0.88)
• FreeRet: MLLMs as Training-Free Retrievers (ICML 2026 · similarity 0.88)
• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.88)
• RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.88)