Research map / Vision-language and multimodal / Cross-modal learning
Vision-Language Models Create Cross-Modal Task Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that autoregressive VLMs map equivalent task specifications (text/image examples, instructions) into a shared task vector, measured via cross-modal activation patching onto image queries.
Problem
How VLMs internally represent task information and whether different modalities (text, image) and formats (examples, instructions) of the same task converge to a shared representation.
Approach
Measures cross-modal alignment by patching a task vector extracted from text examples (or instructions) into an image query forward pass; the single compressed vector outperforms few-shot prompting with the full cross-modal examples, and task vectors transfer from base LLM to fine-tuned VLM.
Key results
- Cross-modal patching of text-example task vectors onto image queries reaches 0.51 average accuracy on Idefics2 versus 0.18 for few-shot prompting with text examples.
- Patching outperforms the strongest unimodal image-example baseline by 8-13% across models.
- Task vectors from base LLM (Vicuna/Mistral) have cosine similarity of 0.89 or more with their fine-tuned VLM counterparts; LLM-VLM patching averages 0.37 on LLaVA vs 0.32 for VLM-VLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Patch the text-example task vector instead of using cross-modal few-shot prompting | LLaVA-v1.5 · Six cross-modal tasks, image queries | accuracy (Avg.): 0.18 -> 0.32 | Helped |
| Patch the text-example task vector instead of using cross-modal few-shot prompting | Mantis-Fuyu · Six cross-modal tasks, image queries | accuracy (Avg.): 0.09 -> 0.31 | Helped |
| Patch the text-example task vector instead of using cross-modal few-shot prompting | Idefics2 · Six cross-modal tasks, image queries | accuracy (Avg.): 0.18 -> 0.51 | Helped |
| Patch the text-example task vector rather than use image-example few-shot prompting | Idefics2 · VQAv2-derived extended VQA tasks | accuracy (Avg.): 0.52 -> 0.69 | Helped |
| Patch an instruction task vector to override the original task rather than add a system prompt | Idefics2 · Semantic task overriding | override success rate: 0.09 -> 0.36 | Helped |
| Patch an instruction task vector to override the original task rather than add a system prompt | Idefics2 · Syntax task overriding | override success rate: 0.49 -> 0.59 | Helped |
| Patch an instruction task vector to override the original task rather than add a system prompt | Idefics2 · Creative Generation task overriding | override success rate: 0.06 -> 0.65 | Helped |
| Patch an instruction task vector to override the original task rather than add a system prompt | Idefics2 · Factual Recall task overriding | override success rate: 0.15 -> 0.42 | Helped |
| Replace long-context prompting with task-vector patching | Cross-modal patching · N=30 text examples, forward pass | runtime (seconds): 2.20 -> 0.20 | Helped |
| Derive the task vector in the base LLM rather than the VLM (ablation) | LLaVA-v1.5 · Six cross-modal tasks, image queries | accuracy (Avg.): 0.32 -> 0.37 | Helped |
| Derive the task vector in the base LLM rather than the VLM (ablation) | Idefics2 · Six cross-modal tasks, image queries | accuracy (Avg.): 0.51 -> 0.52 | Helped |
| Use image examples rather than text examples to derive the vector for text queries (ablation) | Idefics2 · Six cross-modal tasks, text queries | accuracy (Avg.): 0.65 -> 0.42 | Hurt |
Surprising findings
- Few-shot prompting with cross-modal text examples on image queries performs near the No Context baseline, with models often captioning rather than applying the task.
- Patching task vectors from base LLM to VLM performs 1-5% better than VLM-only patching.
Limitations
- Lacks a definitive explanation for why cross-modal task representations emerge.
- Evaluation limited to six synthetic tasks plus a small VQA extension and three VLMs.
- Image examples underperform text examples partly because they require an implicit recognition step.
Experimental setup
Datasets: VQAv2 LLaVA-ReCap OK-VQA A-OKVQA
Benchmarks: Country-Capital Country-Currency Animal-Latin Animal-Young Food-Color Food-Flavor VQAv2 extended tasks task overriding
Metrics: accuracy (first-token exact match) GPT-4o rated overriding success cosine similarity
Baselines: Few-shot prompting (text examples) Image examples patching Image examples few-shot prompting No Context System prompt
Builds on: task vectors activation patching function vectors logit lens
Closest papers by meaning
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.92)
- Understanding Task Transfer in Vision-Language Models (CVPR 2026 · similarity 0.90)
- LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.90)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.90)
- Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability (NeurIPS 2026 · similarity 0.89)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.