Research map / Detection and segmentation / Open-set and few-shot detection
Towards Global Optimal Visual In-Context Learning Prompt Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Partial2Global, a transformer-based list-wise ranker plus a Hodge-style consistency-aware ranking aggregator to select in-context examples for visual in-context learning built on MAE-VQGAN, reaching SOTA on segmentation, detection and colorization.
Problem
Selecting the best in-context example (prompt) for each query in visual in-context learning is a ranking problem; prior pairwise contrastive selection (VPR) underuses ranking observations and yields no globally consistent ranking.
Approach
A transformer list-wise ranker scores k alternatives jointly with the query (class and patch tokens from DINOv2), trained with margin, NeuralNDCG and MSE losses; partial rankings from shuffled subsets are aggregated by least squares over pairwise preference matrices to yield a globally consistent ranking.
Key results
- Without voting, Partial2Global reaches 38.40 average segmentation mIoU vs 35.56 for SupPR (Pascal-5i), 30.66 detection mIoU vs 28.22, and 0.58 colorization MSE vs 0.63.
- With voting, segmentation mIoU reaches 42.69 vs prompt-SelF 41.02, and detection 32.52 vs 29.83.
- Oracle best-of-50 IoU is roughly 49-53 per fold, showing substantial headroom remains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SupPR's pairwise contrastive ranker with a list-wise ranking model (ablation) | Visual in-context example selection with MAE-VQGAN · Pascal-5ᶦ segmentation, average | mIoU: 35.56 -> 37.10 | Helped |
| add the consistency-aware ranking aggregator to rank-10 predictions (ablation) | Partial2Global · Pascal-5ᶦ segmentation, average | mIoU: 37.10 -> 38.09 | Helped |
| combine rank-5 and rank-10 predictions with the aggregator (ablation) | Partial2Global · Pascal-5ᶦ segmentation, average | mIoU: 38.09 -> 38.40 | Helped |
| add test-time voting | Partial2Global · Pascal-5ᶦ segmentation, average | mIoU: 38.40 -> 42.69 | Helped |
| replace DINOv2 with DINOv1 (ablation) | Partial2Global with consistency-aware aggregation · Pascal-5ᶦ segmentation, average | mIoU: 38.40 -> 39.19 | Helped |
| select the second- or third-ranked example instead of the top-ranked example (ablation) | Partial2Global · Pascal-5ᶦ segmentation, average | mIoU: 38.40 -> 38.19; 38.40 -> 38.25 | Hurt |
| fuse predictions from multiple top-ranked examples (ablation) | Partial2Global · Pascal-5ᶦ segmentation, average | mIoU: 38.40 -> 39.13; 38.40 -> 39.73; 38.40 -> 39.77 | Helped |
| remove L_sort from the ranking loss (ablation) | Partial2Global on colorization · ILSVRC2012 colorization | MSE: 0.583 -> 0.601 | Hurt |
| remove L_margin from the ranking loss (ablation) | Partial2Global on colorization · ILSVRC2012 colorization | MSE: 0.583 -> 0.595 | Hurt |
| remove L_mse from the ranking loss (ablation) | Partial2Global on colorization · ILSVRC2012 colorization | MSE: 0.583 -> 0.585 | Helped |
| set margin δ to 0 instead of the selected setting (ablation) | Partial2Global on colorization · ILSVRC2012 colorization | MSE: 0.594 | Hurt |
| vary alternative set size from 50 to 25 or 100 (ablation) | Partial2Global · Pascal-5ᶦ segmentation | mIoU: 25: 38.48 / 41.82 / 37.14 / 35.60; 50 (main): 38.81 / 41.54 / 37.25 / 36.01; 100: 38.81 / 41.80 / 37.90 / 36.00 | Mixed |
Surprising findings
- Visual similarity correlates only weakly with in-context example quality; high-similarity examples often fail.
- DINOv2 performs worse than DINOv1 on segmentation despite being the newer model.
Limitations
- Performance depends heavily on the underlying in-context model, here an early MAE-VQGAN.
- Oracle upper bound shows large gap remaining to the best achievable example selection.
- Cross-fold transfer shows the per-fold training data may be insufficient for a robust ranker.
Experimental setup
Datasets: Pascal-5i Pascal VOC 2012 ILSVRC2012
Benchmarks: Pascal-5i Pascal VOC 2012 ILSVRC2012
Metrics: mIoU MSE
Baselines: MAE-VQGAN UnsupPR SupPR prompt-SelF
Builds on: MAE-VQGAN Visual Prompt Retrieval (VPR) NeuralNDCG HodgeRank
Closest papers by meaning
- Towards Reliable and Holistic Visual In-Context Learning Prompt Selection (NeurIPS 2025 · similarity 0.97)
- Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.91)
- Test-Time Visual In-Context Tuning (CVPR 2025 · similarity 0.90)
- Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning (CVPR 2025 · similarity 0.89)
- Visual In-Context Prompting (CVPR 2024 · similarity 0.88)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.