Research map / Vision-language and multimodal / Human and embodied modeling

Embracing Collaboration Over Competition: Condensing Multiple Prompts for Visual In-Context Learning

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Scene graph generation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CONDENSER, a lightweight plugin that condenses multiple retrieved visual in-context prompts into one fine-grained prompt, trained end-to-end with a frozen MAE-VQGAN backbone.

Problem

Prompt selection in visual in-context learning (VICL) assumes a single ideal prompt exists in the candidate pool; in practice multiple partially suitable prompts exist and selecting one discards useful context, while downsampling and output-ensemble alternatives lose detail or are costly.

Approach

Instead of competitively choosing one prompt, a patch-wise cross-attention module fuses K retrieved image-label prompts at the input level into a single condensed prompt that matches the query's spatial layout, trained end-to-end with the frozen backbone via token prediction and a pre-alignment loss.

Key results

• CONDENSER with K=16 prompts reaches 46.63 mean mIoU on Pascal-5i segmentation vs 43.14 for InMeMo (about 8% relative gain at K=1 setting comparisons reported in paper).
• Single-object detection mIoU rises to 44.64 with K=16 versus 43.21 for InMeMo; colorization MSE 0.54 vs 0.58 for Partial2Global.
• Inference time grows only from 59.17 to 66.61 ms/query when going from K=1 to K=16, whereas Prompt-SelF with K=16 takes 989.62 ms/query.

Measured effects reported in the paper

Surprising findings

• Full cross-attention between query and prompt patches performs worse than removing query-prompt interaction entirely.
• Pre-alignment loss alone is insufficient; removing token prediction loss causes severe degradation (e.g., 8.66 mean mIoU at K=16).

Limitations

• Higher GPU memory cost than single-prompt baselines (1021.86 MB vs 413-565 MB at K=16).
• Pre-alignment acts only as a regularizer; the method depends on end-to-end backbone feedback, so zero-shot or backbone-agnostic use is not shown.
• Gains depend on retrieval quality; weaker retrievers reduce performance.

Experimental setup

Datasets: Pascal-5i Pascal VOC 2012 ImageNet-1K COCO-5i

Benchmarks: Pascal-5i Pascal VOC 2012 ImageNet-1K colorization

Metrics: mIoU MSE inference time (ms/query) GPU cost (MB/query)

Baselines: Random selection UnsupPR SupPR Prompt-SelF Partial2Global InMeMo LoRA P-Tuning V2

Builds on: MAE-VQGAN Prompt-SelF Partial2Global InMeMo VPR

Closest papers by meaning

• Exploring Task-Level Optimal Prompts for Visual In-Context Learning (AAAI 2025 · similarity 0.92)
• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.90)
• Towards Global Optimal Visual In-Context Learning Prompt Selection (NeurIPS 2024 · similarity 0.89)
• VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.89)
• Test-Time Visual In-Context Tuning (CVPR 2025 · similarity 0.88)
• IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)