Research map / Vision-language and multimodal / Multimodal LLMs

Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Task Preference Optimization (TPO), which attaches learnable task tokens and task-specific heads (region, temporal, mask) to an MLLM and co-trains them with conversation data in a 3-stage scheme, improving VideoChat2 on multimodal benchmarks and enabling zero-shot grounding, tracking, and segmentation.

Problem

MLLMs lack fine-grained, precise visual understanding, and existing methods that teach visual tasks as text (autoregressively) improve task performance at the cost of general multimodal ability.

Approach

Treat visual task annotations as differentiable preferences: learnable task tokens pass MLLM hidden states to dense task heads (region MLP, CG-DETR-based temporal head, SAM2-based mask head), so gradients from dense task losses flow back into the MLLM, trained with a 3-stage local-to-global scheme.

Key results

• VideoChat-TPO reaches 66.8 average on MVBench vs 60.4 for VideoChat2 (+6.4), and 40.2 on MMIU vs 35.0 (+5.2).
• On Charades-STA zero-shot, VideoChat-TPO gets R@0.5 of 40.2 vs 33.0 for ChatVTG (previous MLLM SOTA).
• Applying TPO to LLaVA-OneVision raises MVBench from 56.7 to 64.8 (+8.1).

Measured effects reported in the paper

Surprising findings

• Textualizing task data into QA form does not improve, and can slightly hurt, MVBench performance versus the baseline (64.8 vs 66.8 for TPO), suggesting dense supervision is needed.
• A simple two-layer MLP temporal head performs much worse on Charades-STA (R@0.5 17.8 vs 40.2) than CG-DETR, though it still improves MVBench over no-TPO baseline.

Limitations

• Focuses only on discriminative visual tasks, not generative ones.
• Framework is supervised by human annotations, not unsupervised or self-supervised learning.
• Scalability in task diversity and annotation requirements is restricted.

Experimental setup

Datasets: MeViS SAMv2 ActivityNet TACoS QVHighlight DiDeMo QuerYD HiREST NLQ Allseeing-V2 Visual Genome RefCOCO

Benchmarks: MVBench VideoMME MLVU NExT-GQA MMIU SEED-Bench2 Charades-STA QVHighlight RefCOCO LaSOT GOT-10k Ref-YouTube-VOS

Metrics: accuracy mIoU R@1 IoU IoU J&F success rate mAP HIT@1

Baselines: VideoChat2 TimeChat ST-LLM VideoLLaVA LLaVA-Interleave InternLM-XComposer ChatVTG HawkEye NExT-Chat Shikra VideoLISA UniVTG

Builds on: VideoChat2 CG-DETR SAM2 LoRA LLaVA-OneVision

Closest papers by meaning

• Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs (NeurIPS 2025 · similarity 0.91)
• MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (ICLR 2025 · similarity 0.91)
• TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types (ICLR 2025 · similarity 0.91)
• OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
• Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.90)