Research map / Vision-language and multimodal / Multimodal LLMs

Understanding Task Transfer in Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies zero-shot cross-task transfer in VLMs by LoRA-finetuning Qwen-2.5-VL (3B/7B/32B) on each of 13 BLINK perception tasks and measuring effects on other tasks with the Perfection Gap Factor metric.

Problem

Finetuning a VLM on one visual perception task can unpredictably change its zero-shot performance on other perception tasks, and existing accuracy-gain metrics ignore differences in task difficulty and ceilings.

Approach

Introduces the Perfection Gap Factor (PGF), the ratio of accuracy gain to remaining gap to a ceiling, and builds on it breadth-and-magnitude-aware transferability and malleability scores, a task transfer graph, cliques, and task personas (donors, pirates, sponges, sieves).

Key results

• Positive task transferability increases with model size across Qwen-2.5-VL 3B, 7B, and 32B.
• Low-level tasks (Relative Depth, Relative Reflectance, Visual Correspondence) show the highest positive transferability and malleability.
• Qwen-2.5-VL 32B has a maximal positive task clique of size 9; PGF-informed data selection matches or exceeds direct finetuning on Jigsaw and Object Localization in preliminary experiments with 7B.

Surprising findings

• PGF-informed data mixtures outperform direct target-task finetuning on Jigsaw and Object Localization.
• Negative transfer does not show a consistent trend with model size, unlike positive transfer.

Limitations

• Analysis relies on multiple-choice benchmarks, which may suppress open-ended failure modes.
• Only three Qwen-2.5-VL variants plus one LLaVA model were studied; newer models not covered.
• Data selection experiment is preliminary and limited to Qwen-2.5-VL 7B.

Experimental setup

Datasets: BLINK DreamSim (Nights) TallyQA Depth in the Wild TARA WikiArt FunKPoint SPair-71k Visual Spatial Reasoning LVIS HPatches Wild 6D

Benchmarks: BLINK VSI-Bench

Metrics: Perfection Gap Factor transferability malleability accuracy

Baselines: GPT-4o GPT-4V LLaVA-1.5-13B

Builds on: LoRA QLoRA Taskonomy

Closest papers by meaning

• Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.90)
• Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.90)
• Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.89)
• Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.89)
• Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
• VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.89)