Research map / Vision-language and multimodal / Multimodal LLMs

Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a SigLIP-based vision encoder to take text instructions as input and produce instruction-conditioned embeddings, using LLM-generated ImageNet image-query-answer triplets, and reports gains on MMVP-VLM, GQA and instructive retrieval benchmarks.

Problem

Vision foundation models produce static, query-agnostic embeddings, pushing task adaptation onto large downstream VLMs, which is costly and prone to hallucination on fine-grained details.

Approach

Inject text-query embeddings (from a frozen SigLIP text tower, via a linear projection) into the vision transformer so the image embedding is steered by the instruction, trained with a sigmoid alignment loss on Gemini-generated image-query-answer triplets distilled from an LLM.

Key results

• On MMVP-VLM, SigLIP ViT-SO-14 (Ours) reaches 76.3% average zero-shot accuracy versus 42.0% for BRAVE, a 34-point gain.
• On GQA retrieval, Ours (ViT-B-16) achieves 71.2% top-1 versus 63.3% for LLaVA with far fewer parameters.
• On instructive ImageNet† retrieval, SigLIP 2 ViT-SO-14 improves from 38.03% (SigLIP) to 87.06% (Ours), though still below the Gemini oracle.

Surprising findings

• Replacing specific queries with neutral 'caption the image' prompts collapses GQA accuracy from 67.4 to 13.1, showing strong dependence on query specificity.
• Training with a neutral query and rich answers gives high MMVP accuracy (65.1) while GQA collapses, indicating benchmarks respond very differently to conditioning.

Limitations

• Performance depends heavily on query phrasing, requiring empirical tuning.
• Pretrained text encoder struggles with compositional, negated, or abstract queries.
• Training code and dataset are not released due to internal infrastructure and review; comparisons to LLM-based baselines use different evaluation protocols and the Gemini-based test sets are self-generated.

Experimental setup

Datasets: ImageNet CC3M-VQA Open Images WebLI MMVP-VLM GQA Caltech101 SUN397 RefCOCO VQAv2

Benchmarks: MMVP-VLM GQA ImageNet† Caltech 101† SUN† RefCOCO† ImageNet OCR typographical attack ImageNet

Metrics: top-1 retrieval accuracy zero-shot accuracy

Baselines: SigLIP Fusion Menon et al. LLaVA InstructBLIP BRAVE BLIP-2 DFN MetaCLIP EVA02

Builds on: SigLIP SigLIP 2 CLIP Menon et al. (descriptive text augmentation for retrieval)

Closest papers by meaning

• LaViP: Language-Grounded Visual Prompting (AAAI 2024 · similarity 0.90)
• LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models (CVPR 2026 · similarity 0.89)
• NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning (NeurIPS 2026 · similarity 0.89)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.89)
• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.89)
• Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning (ICML 2024 · similarity 0.89)