Research map / Vision-language and multimodal / Human and embodied modeling
Learning What Helps: Task-Aligned Context Selection for Vision Tasks
Jingyu Guo, Emir Konuk, Fredrik Strand, Christos Matsoukas, Kevin Smith
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn task-aligned context selection instead of frozen DINO similarity retrieval | ViT-S/16 classifier · 11 fine-grained datasets (average) | accuracy: 86.6 -> 88.3 | Improved |
| learn task-aligned context selection instead of frozen DINO similarity | ViT-S/16 classifier · 4 medical classification datasets (average) | average score: 92.0 -> 92.9 | Improved |
| learn task-aligned context selection instead of frozen DINO similarity | ViT-S/16 with gated cross-attention · Kvasir-SEG | IoU: 80.0 -> 81.1 | Improved |
| pair query with random context image | ViT-S/16 classifier · Fine-grained average | accuracy: 86.3 -> 86.0 | Hurt |
| provide blank image as context (ablation) | TACS architecture · 15 classification datasets | average score: 87.6 -> 87.5 | No clear effect |
| provide duplicate of query as context (ablation) | TACS architecture · 15 classification datasets | average score: 87.6 -> 87.7 | No clear effect |
| provide noisy query as context (ablation) | TACS architecture · 15 classification datasets | average score: 87.6 -> 87.8 | No clear effect |
| use frozen DINOv3 similar-image retrieval as context (ablation) | TACS architecture · 15 classification datasets | average score: 87.6 -> 88.0 | Improved |
| add learned TACS selection over frozen similarity retrieval (ablation) | TACS architecture · 15 classification datasets | average score: 88.0 -> 89.5 | Improved |
| add policy-gradient path to differentiable selection (ablation) | TACS selector · Fine-grained classification | average score: 86.7 -> 88.3 | Improved |
| add Gumbel-Softmax differentiable path to policy-only selection (ablation) | TACS selector · Fine-grained classification | average score: 87.2 -> 88.3 | Improved |
| differentiable selection only vs frozen retrieval (ablation) | TACS selector · Medical classification | average score: 92.0 -> 92.1 | No clear effect |
Closest papers by meaning
- L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement (ICLR 2025 · similarity 0.86)
- Task-Aware Clustering for Prompting Vision-Language Models (CVPR 2025 · similarity 0.86)
- Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning (CVPR 2025 · similarity 0.85)
- Retrieving Counterfactuals Improves Visual In-Context Learning (CVPR 2026 · similarity 0.85)
- Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment (CVPR 2025 · similarity 0.84)
- TTRV: Test-Time Reinforcement Learning for Vision Language Models (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
