Research map / Vision-language and multimodal / Human and embodied modeling
Towards Reliable and Holistic Visual In-Context Learning Prompt Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RH-Partial2Global, which filters VICL in-context candidates with a jackknife conformal prediction reliability test and replaces random sampling in Partial2Global with covering-design-based sampling, improving segmentation, detection, and colorization results.
Problem
In-context example selection for visual in-context learning (VICL) relies on a similarity-priority assumption and random sampling of pairwise preferences, which may be unreliable and incomplete.
Approach
Use jackknife conformal prediction to keep only candidates whose prompt quality is consistent with their visual similarity, and replace random shuffling in Partial2Global's local ranking with a precomputed covering design so every candidate pair is compared with minimal redundancy.
Key results
- RH-Partial2Global reaches 39.02 average mIoU on Pascal-5i segmentation versus 38.40 for Partial2Global.
- On single object detection RH-Partial2Global gets 30.94 mIoU vs 30.66 for Partial2Global; colorization MSE improves from 0.58 to 0.56.
- With test-time voting, RH-Partial2Global+voting reaches 43.08 average segmentation mIoU vs 42.69 for Partial2Global+voting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Partial2Global with RH-Partial2Global, adding conformal candidate selection and covering-design sampling | Partial2Global · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 38.40 -> 39.02 | Helped |
| Replace Partial2Global with RH-Partial2Global, adding conformal candidate selection and covering-design sampling | Partial2Global · Pascal VOC 2012 single-object detection | mIoU: 30.66 -> 30.94 | Helped |
| Replace Partial2Global with RH-Partial2Global, adding conformal candidate selection and covering-design sampling | Partial2Global · ILSVRC2012 colorization | MSE: 0.58 -> 0.56 | Helped |
| Replace Partial2Global+voting with RH-Partial2Global+voting | Partial2Global+voting · Pascal-5ᶦ segmentation, Avg. (mIoU), voting | mIoU: 42.69 -> 43.08 | Helped |
| Replace Partial2Global+voting with RH-Partial2Global+voting | Partial2Global+voting · Pascal VOC 2012 single-object detection, voting | mIoU: 32.52 -> 33.28 | Helped |
| Add conformal prediction-guided candidate selection (ablation) | Partial2Global · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 38.40 -> 38.78 | Helped |
| Replace random sampling with covering design-based sampling (ablation) | Partial2Global · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 38.40 -> 38.79 | Helped |
| Add conformal selection and covering design sampling together (ablation) | Partial2Global · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 38.40 -> 39.02 | Helped |
| Add covering design sampling to conformal selection (ablation) | Partial2Global with conformal selection · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 38.78 -> 39.02 | No clear effect |
| Add auxiliary filling with the most similar candidates from the reliable set (ablation) | RH-Partial2Global · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 39.02 -> 39.27 | Helped |
| Filter initial candidates using conformal selection (ablation) | Initial alternative set · Pascal-5ᶦ segmentation, average top-5 oracle IoU | IoU: 46.72 -> 46.46 | Hurt |
| Use Spearman correlation instead of negative KL Divergence as the conformity function (ablation) | RH-Partial2Global · Pascal-5ᶦ segmentation, Avg. (mIoU) | mIoU: 39.02 -> 38.77 | Hurt |
Surprising findings
- Spearman correlation between visual similarity and in-context IoU is statistically significant for most queries, yet the average correlation coefficients are very low (about 0.03 to 0.05), indicating weak similarity-performance association.
- Applying conformal selection and covering design together did not give additive gains; an auxiliary filling step was needed to restore improvement.
Limitations
- Reliable set prediction is sensitive to dataset size; gains are smaller on folds with fewer samples (Fold-0 and Fold-3).
- Improvements over Partial2Global are modest (about 0.6 mIoU average on segmentation).
Experimental setup
Datasets: Pascal-5i Pascal VOC 2012 ILSVRC2012
Benchmarks: Pascal-5i Pascal VOC 2012 ILSVRC2012
Metrics: mIoU MSE
Baselines: MAE-VQGAN UnsupPR SupPR Partial2Global prompt-SelF
Builds on: Partial2Global VPR MAE-VQGAN jackknife conformal prediction
Closest papers by meaning
- Towards Global Optimal Visual In-Context Learning Prompt Selection (NeurIPS 2024 · similarity 0.97)
- Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.90)
- Test-Time Visual In-Context Tuning (CVPR 2025 · similarity 0.89)
- Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning (CVPR 2025 · similarity 0.88)
- In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.86)
- Retrieving Counterfactuals Improves Visual In-Context Learning (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.