Research map / Architectures and efficiency / Pruning and efficient tuning
Exploring Task-Level Optimal Prompts for Visual In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that most test samples share the same best demonstration prompt in visual in-context learning, and proposes training-free task-level prompt search (Top-K and Greedy) that cuts prompt-search cost while matching or beating sample-level methods.
Problem
Finding optimal demonstration prompts for each test sample in visual in-context learning (VICL) is computationally expensive and prone to overfitting when reward models are trained on small labeled sets.
Approach
Observes that the best-performing demonstration set is largely shared across test samples within a task, so it searches a single task-level prompt over a labeled set using training-free Top-K (rank single demonstrations) and greedy (add demonstrations with early stopping) strategies.
Key results
- Greedy task-level prompt reaches 36.76 average segmentation mIOU versus 34.61 for the best sample-level baseline (SupPR-SelF), and matches Oracle on colorization MSE (61.56).
- Greedy is within about 3% of Oracle on detection (28.25 mIOU) and about 6% on segmentation average (38.45 Oracle).
- More than 27% of test samples attain their best performance under the single task-level prompt, versus 15.03% under sample-level selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use task-level Greedy prompt selection instead of UnsupPR | MAE-VQGAN · Pascal VOC 2012 single object detection | mIOU: 24.01_{0.53} -> 28.25_{2.94} | Helped |
| Use task-level Greedy prompt selection instead of prompt-SelF | MAE-VQGAN · ImageNet colorization | MSE: 45.61_{0.36} -> 61.56_{1.27} | Hurt |
| Use task-level Greedy prompt selection instead of sample-level baselines | MAE-VQGAN · Pascal-5i segmentation Split-0 | mIOU: 37.28_{1.19} -> 36.86_{1.38} | Mixed |
| Use task-level Greedy prompt selection instead of SupPR | MAE-VQGAN · Pascal-5i segmentation Split-1 | mIOU: 39.74_{1.78} -> 42.22_{1.42} | Helped |
| Use task-level Greedy prompt selection instead of SupPR | MAE-VQGAN · Pascal-5i segmentation Split-2 | mIOU: 34.24_{1.88} -> 37.11_{0.87} | Helped |
| Use task-level Greedy prompt selection instead of SupPR-SelF | MAE-VQGAN · Pascal-5i segmentation Split-3 | mIOU: 30.40_{0.37} -> 30.84_{1.84} | Helped |
| Use task-level Greedy prompt selection instead of SupPR | MAE-VQGAN · Pascal-5i segmentation Avg. | mIOU: 35.03 -> 36.76 | Helped |
| Use Top-1 instead of Greedy prompt selection (ablation) | MAE-VQGAN · Pascal VOC 2012 single object detection | mIOU: 28.25_{2.94} -> 28.25_{2.94} | No clear effect |
| Use Top-1 instead of Greedy prompt selection (ablation) | MAE-VQGAN · ImageNet colorization | MSE: 61.56_{1.27} -> 61.56_{1.27} | No clear effect |
| Use Top-1 instead of Greedy prompt selection (ablation) | MAE-VQGAN · Pascal-5i segmentation Split-0 | mIOU: 36.86_{1.38} -> 32.41_{2.31} | Hurt |
| Use Top-1 instead of Greedy prompt selection (ablation) | MAE-VQGAN · Pascal-5i segmentation Avg. | mIOU: 36.76 -> 35.46 | Hurt |
| Increase Top-K from 1 to 2 (ablation) | MAE-VQGAN · Pascal-5i segmentation Avg. | mIOU: 35.46 -> 36.81 | Helped |
Surprising findings
- Using a single task-level prompt for all samples performs better than costly per-sample prompt selection methods.
- Increasing the number of demonstrations (larger K) can degrade detection and colorization performance.
Limitations
- Top-K performance is sensitive to the hyperparameter K, which cannot be reliably set.
- Selected prompts still fail on fine details such as bird legs and wall cracks.
- Evaluated only with MAE-VQGAN and on three tasks; generality to other VFMs is not shown.
Experimental setup
Datasets: Pascal-5i Pascal VOC 2012 ImageNet
Benchmarks: Pascal-5i Pascal VOC 2012 ImageNet
Metrics: mIOU MSE
Baselines: Random Oracle UnsupPR prompt-SelF SupPR SupPR-SelF
Builds on: MAE-VQGAN Painter SupPR prompt-SelF UnsupPR
Closest papers by meaning
- Test-Time Visual In-Context Tuning (CVPR 2025 · similarity 0.92)
- Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning (CVPR 2025 · similarity 0.92)
- Towards Global Optimal Visual In-Context Learning Prompt Selection (NeurIPS 2024 · similarity 0.91)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.90)
- Towards Reliable and Holistic Visual In-Context Learning Prompt Selection (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.