Research map / Vision-language and multimodal / Human and embodied modeling
Finding Visual Task Vectors
Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, Amir Bar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| patch REINFORCE-found Task Vectors (no ICL example) instead of one-shot prompting | MAE-VQGAN · Pascal-5i Low Light Enhancement | MSE: 0.685 -> 0.458 | Improved |
| Task Vector patching on segmentation | MAE-VQGAN · Pascal-5i Segmentation | mIoU: 0.338 -> 0.353 | Improved |
| search patch positions with REINFORCE instead of greedy random search (ablation) | Task Vector search · Pascal-5i In-painting | MSE: 0.580 -> 0.480 | Improved |
| use causal mediation analysis scoring instead of REINFORCE (ablation) | Task Vector search · Pascal-5i Segmentation | mIoU: 0.353 -> 0.230 | Hurt |
| patch Task Vectors in decoder only or encoder only instead of both (ablation) | REINFORCE Task Vectors · Pascal-5i Segmentation split 0 | mIoU: 0.35 -> 0.32 (decoder only), 0.09 (encoder only) | Hurt |
| patch whole attention heads instead of tokens or quadrants (ablation) | REINFORCE Task Vectors · Pascal-5i Low Light Enhancement | MSE: 0.458 -> 0.942 | Hurt |
| quadrant vs token granularity (ablation) | REINFORCE Task Vectors · Pascal-5i, four tasks | mIoU/MSE: Segmentation 0.350 (T) vs 0.338 (Q); Lowlight 0.495 (T) vs 0.458 (Q) | Mixed |
| patch top-scoring quadrants directly without iterative search (ablation) | Task Vector baselines · Pascal-5i Segmentation | mIoU: 0.353 -> 0.150 | Hurt |
| add one in-context example on top of Task Vectors (ablation) | Task Vectors · Pascal-5i Segmentation | mIoU: 0.353 -> 0.346 | No clear effect |
| train a single multi-task patching placement instead of task-specific placements (ablation) | REINFORCE Task Vectors · Pascal-5i Colorization | MSE: 0.453 -> 0.502 | Hurt |
Closest papers by meaning
- Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.89)
- Do different prompting methods yield a common task representation in language models? (NeurIPS 2025 · similarity 0.88)
- Task-Aware Clustering for Prompting Vision-Language Models (CVPR 2025 · similarity 0.88)
- Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation (NeurIPS 2025 · similarity 0.87)
- Query Efficient Black-Box Visual Prompting with Subspace Learning (CVPR 2025 · similarity 0.87)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
